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presented in the next chapter, followed by a summary.c ér. 

'' Material ‘prepared at the conference on the evaluaticn of ‘vocational 


= 2 educatton is peckeeene in an epilogue. (LEAL, ; | 
- . - _ a 
' (Cy ye —. | oe | - 
| a ‘ re 6 8 
= 


ba Bete ett to RG ee ee ee a te MACH HD er rere LPP EPR LEE LET | errr TT 


ee Reproductions supplied by EDRS are the best that can be mde # 
, * -from the original docunent. * 
ERIC | mag (iano vecureserenc ss srempetsvigt tearnonteessseeacessedhces) there 


nan. eae : , tg 4 
ae es . + > “ i ak 


By 
oT 
ee 
a . 
Law . 


“.? 


_ : . : 7 Research and Development Series No.473-) 
ot —* 7 a a 
pe 
. e l'¢ - \ 
Cas 
; N ¢ 7 
EVALUATION OF VOCATIONAL EDUCATION: 
ROLES, RESPONSIBILITIES, AND 
+ RESPONSES OF STATE AND 
FEDERAL AGENCIES , 
‘a : ‘ 
: , 
., a’ 


Prepared by Pr 
\ 


oe ee Education Commission of the States. 
Suite 300, 1860 Lincoln Street 
Denver, Colorado 80295 
i t 


ce 
_ Under Subtontract to | 


‘The Gabiohal Center for Research: in Wouselonal Education 
a, * | 


me The Ohio State University 
-. + 1960 Kenny Road 
' : \ _Columbus,,. Ohio 43210 


° ° AR 7 
ef » we 1979 
N 
ye . 
4’ . 
\ be 
h 
US.OBPARTMENTOFRHaALTH, * é y 
HOUCATIONA WELFARE. ee ts ; i 


NATIONAL INSTITUTE OF ! 
BDUCATION 


; *, ¢ 
rm THIS DOCUMENT HAS BEEN REPRO Pee ® 2 
DUCHD EXACTLY. AS RECEIVED FROM —. mt , 
© THE PERSON O€ ORGANIZATION ORIGIN: ae 
ATING IT POINTS OF VIEW OR OPINIONS “is a 
STATED OO NOT NECESSARILY RERRAS. = ign 
SENT OFFICIAL NATIONAL INSTITUTE OF a 
HOUCATION POSITION OR POLICY © os 


. 


\ 


toe BORE We te S Po a i 
co iets 5 Mae on r 4 i oy. 
f 7 “e oa oe . Re: ‘ + 
“ vv : : ‘ 4 a +P Sag : 
. ° ‘ ae a _ a 
, : eae Ne. 
ft ~f : nt - : 
a . my . 
‘ ' ’ 7 ’ e 
¥ t 
\ 
we * : : ‘ 
7 ; - 
‘ _4 . . 
. 
— *. ~ 
. > 
. 4. : 
‘ we vs 
’ 
) - 
ry . f . 
4 . Ve 
. i . 
. é Nic 
J a : ar = 
. , . 
‘ 2 
3 
. 
9 
\ 
t 
a 
: ‘ 
e ° oe . 
& ’ * | 
< . « 
. t= a . 
1 
- . 
, ‘ , , 
e \ o-% 
. - a : 
. 
5 « 
ad . 
~~ co 


The National Center Mission Statement 
The National Center for Vocational Education's'mission,.is .- . 
to increase the ability of diverse agencies, institutions, an 
‘and, axganizations to solve educational problems relating ‘to * — 
indfvidual career plannina, preparation, and prparession. 
The National Center fulfills its mission by: / . 


@ : ; . » : . > ‘ 
o Generating knowledge through research . 
; ,O Developing educational programs ‘and products. ; 
o° Evatuatina individual program needs and outcomes oe 
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ae. Installing educational programs and products 


o © Operating information systems and services 
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O Conducting: leadership ‘development and training programs 
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Report. of a Project. Conducted 
"Under Contract No. OE 300780032 
The material in “t] is Uhivesdien was prepared. purspant to a 
/ contract with the Burdau of Occupational and Adult Education, 
U. S.. Office of Education, U. S. Department of Health, Education, 
and Welfare. Contractors undertaking such projects under ,.. 
government sponsorship \are encouraged - ‘to freely express their 
judgment in professional} and technical matters. Points of view 
or opinions do-not, therefore, necessarily represent official 
a Bureau of Occupatignal and Adult. ec ats On: U. S. Ofifice of: 
: Education position | or Pomey, A : 
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a the Nat¥onal Center has “é@nsistently enforsed an¢@’} 

evaluations of .vocational education as ‘essqntial -to improving 

programs and maintaining accountability. | spite this commit- 9... | 
ment t6 evaluatien, there has been a continuing concern that. 9° °°. gs 
various evaluation activities may overlap, ‘duplicate or eveh =~ . 
contradict each other, . Thus ,,-as an /jnitial effort toward .© : : a 
defining and, clarifying roles and responsibilities in evaluating — 
vocational. education, the National Center a iked the Educatién: 4. 

' Commission of the States (ECS) to undertake thie study. The  — 
central purpose of this paper is to describe \the origins, —~ 
assumptions, approaches, and problems of the major evaluation 

_efforts relating to vocational education. | The’.study describes 
over twenty: evalvatians: being undertaken by twélve agencies ~ 
and institutions.- It Vis hoped that this document will enable 
_the reader to understand the range of evaluation’ éfforts being 

! undertaken. , . ¢ - | 
t. : = 
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As a subcontractor, the ECS was encouraged to express its 
"views as a result of the investigation it had undertaken. The 
views do not necessarily reflect the views of. the National Center 

or the Bureau-of Occupational and.Adult.Education, the U. S.’.~ 
Office of Education. : 
Sincere appreciation is extended to Sylvia D. Parker of ECS’ 

for fhe successful actomplishment of a. difficult task under ever 
present resource and time constraints: Grateful acknowledgements 
are offered to her ECS colleagues ‘Carol Anderson and Gene 
Hensley for their assistance. Also, for their efforts in 
initiating and monitoring the study, appreciation is extended 
to N. L. McCaslin,: William Stevenson, and Jerry Walker of 
the National Cénter. Finally, appreciation is’ extended to the 
Bureau of Occupational and Adult Education, U. S. Office of ( ; 
Education for their support of this effort through the contract © 

* for tHe National Center for Research in Vocational Education.’ : 
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“Groups: with Specific Evaluation Responsibilities . ‘ 


‘State Boards -for Vocational. Education. ews 
. National Advisory. Coungil for Vocational 
Education ..°. a 
Bureau of Occupational and Adult Education, 
as USOE--Management Evaluation Review for 
( > = Compliance and Quality . a 
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Nationa{ Center for Education Statistics-- | | 
- Vocational Education Rata System...... 
_ Office of Evaluation and Dissemination’. . oe Aa 
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Groups with Responsibilities Related.to Evaluation : 


“National Center for. Research, in Vocational . 
Education .. . ‘. . : ‘ che ce os 
National and State Ocdupational Information 
Coordinating Committees"... ....e6f « « 
General Accounting Office ,. : eo & 
Bure se of Occupational ‘and Adult Sducatibn i: x 
Coordinating Committee on Research in 
Vocational. Education iS: cae ae cers er Gee aw ee 


Miscellaneous . . 1. w°. eee bee Te ee 
CHAPTER III? IMPRESSIONS /REFLECTIONS / OBSERVATIONS a 
CHAPTER IV: SUMMARY .. . 
‘EPILOGUE a rr 
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“CHAPTER II: OVERVIEW’ OF EVALUATION ACTIVITIES } eee. 


State Advisory Councils for Vocational Education... 


' National Institute Of Education aw we ee 


‘+ Vocational Education’ Act, Séction 104. 


_y boards are separate entities while -others are part * public: 


wy. 


. INTPODUCTION 


The Education Amendments of 1976 :-(P.L. 94-482) were very a 

' prescriptive with regard to the evaluation of vocational! educa- 
tion. State boards of vocational education in each. statle were 
directed to evaluate the effectiveness of all their programs 

. assisted. with. ‘funds under this act, and state advisory oo . 
were directed to monitor ‘the evaluations done: by the state a" 
boards.+ The U. S.-.Office of Education was: mandated t nduct 
‘a review of all the states programs,, and ‘the National Advisory 
Council for Vocational Education was instructed to review the 
federal’ level administration: and operation of votatidnal, 
education programs including the reviews“of the. states" programs. 

_ Many other evaluation activities were stated or implied throughout 

' the amendments, as well as various planning, accountability,. 
— and data. collection. provisions that were. all interrelated. 

a , ~~, 

With the idea that the, evaluation ‘Poaui cements in the 
-leaislation are numerous and the purposes may ‘not be clearly 
understood, this: study ‘was concerned with providino an overview’ 

— the requirements anf the varioys agencies’ charaed with re 

valuation responsibilities. This was done softhat vocational 
enucaners at the local, state, and federal levels might have a 
better understanding of who is ‘requiring what of. whom. 


oi 


on 


There are many different meanings for ‘the. word ‘evaluation, 
*It is defined in this paper as any on-going effort that elicits 
information about, vocatjonal education. for. the purpose of 
documenting, improving, -clarifyino, unders Spar nds or de- 
.termining the worth of vocational. educating. ; 
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whe term state board (og. eeatienat education) is used in 
‘this paper in the generic’sense as the term is used_in- the 

A State boaad is- the 
sdle state agency or board responsible for the administration 
of vocational education programs funded under this act. Since 
‘the -governance structure: varies from state to: state, some state 
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2 ge in the Education Amendments of 1976... ‘An overview of the major... 
- groups and their activities in evaluation is presented. : The . 


of these evaluation efforts in-order :to provide an objective 


many intertwining federal ' evaluation, ‘requirements. . Improved | oo 
would provide. an- opportunity to help further. those relationships.” 


.assioned to the. ‘various agencies was seen as a walyable aid-to . 


' tively their evaluation respons 


provisions of P. L. 94-482 and the related regulations and then: ‘to 


director «and/or staff personnel for each of the other groups with 


no survey: forms were used, nor were sources asked the same 


not identified in this paper.’ .In all cases though, colleah waaay 
came from Jnowscdgeable people in ee aa positions. | ‘3 


“This: ancet eonaseds on ‘nose: Saigon on activities specified 


study was not’ an evaluation of evaluation efforts‘ but an. analysis. 


overview of what was ‘taking place in the fall of 1978. ret | 
/ The Education Soni selon of/the. States (ECS) ‘in Soopera~ *: ' 
tion with the National Center: for. Research in Vocational Educa- “ar 
tion, prepared this paper. The Commission became involved in | - < 
the project because. of the concern ‘expressed | by states about the 


federal-state relations in all ‘areas. of education is amajor ,. 
emphasis of the Commission. “This project. was seen. as one that’. 


The area of evaluation is one of the primary. responsibilities of. _ es 
the National Center, A description ‘of evaluation: responsibilities a 


state and local agencies as bade f work to carry. out’ mora, effece io 4 
bilities. Work was begun on’ the Ba 6 
paper in June 1978,. and completed in January of 1979. 


In order to déacribe the present range’ of eee uation ee a 
activities, it was necessary to first’ review the evaluation: - . 


identify the different groups assigned responsibility. for 4 
evaluation. An information-qgathering eriod. flowed. All. of 
the state’directors of vocational education were contacted © for 
materials on their evaluation activities. . After analyzing those 
materials, extended telephone conversations were held with state 
directors.and/or key State staff with evaluation responsibilities | 
and some -éxecutive directors of state advisory. councils. The 


specific and related evaluation responsibilities mentioned in 

this report were also contacted by telephone or in person. Over 
150 telephone calls were made to gather and verify information. 
No* attempt was ‘made to conduct a survey per 4e3— consequently 


questions. All soltrtes were assured of confidentiality and are 


In order to pxesent a section on‘the congredsional intent | r 
behind the activities of each group with: specific evaluation '" 
sponsibilities, information was gleaned from the Congres- . 
nal Record, committee and hearing reports, and comments: from _ 
those who helped author the. amendments, . 
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In January. 19795": a ‘Souter ence: on ‘the: evalyati‘on: of voca= tet ee 
tional education was held in which .a draft of this. ‘papery: ‘was’ aa 
presented for discugsion, and recommendations wére made related 
to the sluation @f vocational‘ education:, Participants in the ~~ 
conf¢rence’ included four state evaluation’ coordinators, | three...” St 
' state administrators, ‘four executive diréc ‘of state advisory - ae 
‘ councils, a member bf the National: Advisory | Council: fot Voca=..° |.) hap 
tional Education, and an executive: director of the ‘state occupa-' , :: 
‘tional infowhation’. coordinating comni:t pee, - Two sthff members:  _~ 
of the Education,Commission of the States and two. representatives : 
of the National’ €enteér for Research in Vocational Education also... °° 
participated in the. discussion. -In addition to ‘providing . feed- , 
back on the usefulness and accuracy of the ‘report the group > a 
prepared a statement of ¢oncerns ave needs: for: vocational. educg~ oo 
tion evaluation. , : 
‘e! 4 ote . " . Og o 
‘There atetive* Harts to “pide report. Boek coun information 
on vocational education, evaluakion and the Education Amendments 
of 1976 is provided in the first section. An.overviéw of : j Oo 
evaluations and some other related reviews, . studies, and services. Se 
The information on,each of the omgoihg evaluations is organized - 2 ° 
in five parts: (1) legislative basis for the activity, (2) - Pe sd 
assumptions and/or expectations behind the legislation and the- . ae 
'. evaluation activity,. (3). approach/activities: (what the group is & 
actually doing), (4) principal. problems, if any, and (5) comments.: © ra 
Impressions, reflections,- and observations are presented th. the . ‘ 
next section followed by a summary. | “The mate ial ‘prepared at os 
the conference on the evaluation of “‘Yecational education: is | 
presenred as an epilogue. ; 
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“« 
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. This report was developed with the. expectation that: infor-’ ons, 
mation of this type will help improve vocational education — 
policies and programs. While much more needs to be'done in the 
area of evaluation of vocational education, ‘the information in) 
this report represents an:important step towards becoming more 
informed abou vocational education ee | oe 
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_ProxgaoonD | 


Ai a Shae the. Smith-Huches Gecueicaal ence lon, Act passed 
following over ten years of ldbbying and deliberation. This 
oe Law reflected: the times: concérn with the opportunities, for.- 
- «the sons and daughters of working people and ‘farmers,’ few of | - 
_.°, whom would goon to college. or, tniversity;.or even finish: high. ae 
°°.” schol; concern with: youth unemployment and urkan’ crime; concern 
with the industrial and agricultural’ production. superiority” ‘of oa 
the German Kinadom of Bavaria; concern that: academically. a 
| ree were could not+respond tothe increased demand for trained {* 
in ustrial and technical workers brought about -by rapid ‘in- 
dustrialization; concern that schools,: if left to ‘their own, 


would not respond to the need for education for work.! Congress _ Las 
‘thus passed the act to encourage the nation's ‘schools to inclide. 7 % 
preparing students. for earning a living as sn. intearal<part of 9 f- 


yp their mission. This was landmark legislation — that pointed ae 
| : toward a major Federal role in education. ira 


a 


- Soyé'igtates suck as Massachusetts ahd some, cities’ had already‘, 

i “> a, days Lolaegy systems of.free industrial educations. Within a year of | -. 
ie ‘thé pass e-of the Smith- Hughes ener that Act, all the ~*~ 
. states had formally accepted. the act and pyp ears 

“*. up for agricultural, ‘trade and. industrial,. aes atone conomics -=-* 
‘' training. ‘Over the years,.more occupational categories for. 
training were added and federal authorization levels and state. 
and local expenditures were aaeiai ly anereaneds”. aa 
a mgt ASR : . 
ba _. The Vocational Education Ack, (VEA) of 1963 brought a change. 
“g in aqoals, an expanded role, and increageds federal funding. - 
Reflection on the 1963 Act by Evans in a report onthe 1968— 5 
ene aa that the: act "BOCURES on the ts tad we needed 


_ 3 ~ .. _— * e+ : i | ) 
oo. _ ‘ : : . : ee Wa : % . we _f . 
eee 1 Albert H. Quie, “Education. for Work -,A National Perspective". - oe 
es (Speech delivered dt the Bicentennial Conference on Vocational ——* 
- “ Educatdon sponsored by the.National, Advisory Council on Voca-’ 
tional Education, in Minneapolis, Minnesota,’ October 1976). ~ tee 


. J i . fe ; ee o ; _ é ag - . f : . : ‘ > : . 
& s ; ; a we 7 | . ae 8 
" . ‘ a 4 : , eae . i 
Vi 7 Ss te . s ; . - . a : . see 
‘ . ex : : Vs : Sy at : : ba ' : ee 
. = . * ig 4 a € " F Py 


skills. oe han Wiecsedans: ae which, needed papi. er 


Thus, instead_o¥ ‘just providing training ‘in selected occupations, 


vocational @ducation was to maintain, extend, and improve exist- 


'.ing programs and develop new programs so that persons of all: 


_ages in all communities would have’ ready accéss to. high eee 
‘traintng or retraining. Vocational education was also to be- 
come’ more responsive to. Lee a with special difficulties that - 


' prevented them-from succe 


eg ‘in regular vocational programs...) .°. 
Development of human potertty and — term enelonee were. ; a 
ene ety expressed sea . : oe 
i a oe wes? Hh Se 
In (1968, phe: fe ae the Vocationa i Education Act. — 
emphasized the need to offer training that was more responsive 
to the needs of people and the changing labor: ‘market. . Provisions 


‘for. categorical funding by service areas: was not the approach. . 


‘taken. in this legislation. Furds were earmarked for training ~ . | ~ 
certain groups including the. disadvantaged: and handicapped,’ and 
postsecondary students’; .Requirements for strengthened planning 


. evaluation - were included. State ay councils for. each. 


ate were also reduired. 7 ae 


aa 
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“4 rinally, in 1976, Aber two General ‘Aesounting. Office 


. reports on vocational education and two years of Congressional 


hearings, the Education Amendments of 1976 were. passed. | As. 
might be expected in the nearly sixty years. since the Smith-. 
Hughes Act, thetconcerns behind the legislation had changed 


somewhat. In ‘1976, youth unemployment was still a major concern. 


Byt inflation had made the need to account for every federal 
dollar spent of major importance. Local and state education a 
agencies had responded to the nee@,for vocational training, ‘and 

by 1974 state and local expenditures , for vocational’ education 

had reached nearly $3 billion,. which was more than 86 percent, ( 
of the total. There was clearly. no need to worry about the public: 


* gchools' willingness to provide vocational education, but there 
@was still concern that special sub-populations did not shave equal - 


. access to training. The role of women in our society and the 


“', quest for equality was an intportant factor causing the Congress 


+ roe 
‘ast 2 P 


k if: vocational education cauld kelp chatde the biases 
na stecectyeing of society? The findings of the Commission on 
National Aid ‘to Vocationa] Education in 1914 that “vocational 
training. was! imperative. to oe the waste of the nation's 
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"2 epee NL ‘Evans etal... Education oe ietoyacne? “The Baak- 

2 and Potenttal off. the 1968 :Voeational-Education Amenid- 
ents (Ann Arbor, Michigan: Institute of Laboy and Industrial 
elations, 1969), p. B.. Jove ae ame ae . 
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cp ernaiy pertinent. in a ae 
course, say-much about what vocational education is. .° People 
of farmers. Wthers 


Still others think that vocational. education is a tracking system 


tated to the preparation of individuale for patd or unpaid. 
employment, -or for addittonal. preparation for a career requiring 
other than a baccalaureate or'advaneed dégree,4 “Wocational:  ~° |: 


ren and adults across:the cofintry. Courses are offered in 
everything from aviation electronics to X-ray technology. 


‘it has ‘been designed to 
Its governance, appearance and,efficacy are not exactly aTike 
in any two. states. From the national level, vocational educa-,, 


4 Sources; economic ‘and social: policies. Vocational education is 


= 3 4 spoeraa Ae vlew ae Seeandary Vocattonal PF sats. oe Ae 


a . | Se ee ee a oe eg = : 
“human and’ natural résources; to. demogratize education, . and to: ee 
enhance, standards of living. and quash Boerat unrest "3 wereistill. oie os 


This brief. history of fetieral ‘tegistartillMoes | not, of . ies 


have “different views of .vocational education. There are those.who. - 
dismiss vocational . education ‘as, education only for ‘the sons _ \ 

& efrn the few: proprietary schools that once Ces, 
of fered: training tq specting students for non-existent jobs, - = * 
and they suspect that all vocational education. must -be .like that.” eS ee 


for the disadvantaged and minorities. And’ there are those oe 
think that vocational education was. the best rand. only. good change. 
they ever got out of - BEhgots 
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The federal tes aacios savines the term as sntaiahoalee ae 
tion as. organized educationdl-programs whieh ire divesciy rex > - 


education is, in fact, a large, diffuse, complex, multi-billion: © 7°. | 
dollar enterprise that ‘provides. work-related training to child~ 


‘Because vocational eaucay os is: part of the education system,» 
€ responsive primarily to local needs. 


tion is also-seen as an, important factor in national.human rer. ~ 7 ee 


Gaught between the world of education and work and sometimes finds _ 

difficulty ‘in being accepted by either. It is sometimes regarded — 
-a stepchild and is often™expected to live up to. the cope hacer 

ng expectatrons anc goals of different groups. 
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Job Weucement and State Funding (Prepared by the Ohio’ negies : 
‘lative Service Commission, Staff Research Report No. 126, , & 
Columbus, Ohio, April 1978) p.3. referéneed from J. Chaster : . 
‘Swanson, Development of Federal Legislation. ae Voeattonal | . 
Education" (Chicago, pene ye: p. 35., Hh tte “ee 


U.S. Congregs,: A bani tation. a, podexat Wao on Lave ag 
Amended Through June 30, 1977, Nocational Education Act, pre- 
pared for the use of the House Committee on Education & Labor 
and the’ Senate Committee on Human Resources mre U.5s. 
soverToment es Office, ae ae pone 


.* 


‘oe ae a ee 
ar hetd by ‘the. Congressighal.‘sub=".. °° i: 


Education Amendment. 020000. 
mea]-education had madd...) 


: _ prevailing economic and.social needs... ,° oe o.. 


ay. OF 


BY 


an . ee a v. Ss a ve iS ; : . ote ay we oo 
. The. amendments that ‘were passed provided. a statement } f.what' . ° 
Congress ‘fdlt' vocational education must: become: °(1) more re~  .”«. 


. sponsive,to the labor inarket, (2) more accountable for how)... = %'s 

: federal doldars are spent, (3) more open.to special sub- /. 9 09: 
‘.. populatigns--women, minorities, handica ped, disadvantage@,  ..-,- 
people with limit@d-English speaking ability, displaceg/home- ." - 


| Makers, “and (4) gless sex biased and stereotyved. Proefams were to — . | 
improve through’an’ increased emphasis on plannyfgr ard evaluation. ..-,  - 
= -- oy ree ss 
_ Réquirements for evaluation and accountability were in-)!.%"." 
.serted -throughout the legislation and reflected the .Conyressiobnal © 
belief that they should provide encduragement to the Exéoutive -— 
Branch;’.and stdte and local agenciey-in order to change: and ~ 
review vocational. education. yring. the Tongressional hearings,:. 
uestions and criticisms had beenfaised with regard to the ©. °° 
ederal. administration of and States’ .compliapce with various ~ 
requirements in the Vocational Education Act.?>- =~! ¥' 


In retragspect, some of the ‘requirements seam nécessary, — | 
some duplicative, and still others contradictory. -For example, . ;- 
in saeking to haye vocational education’ become mare responsive’ ~~ 
‘to the labor market, Congress recognized that ter. labor 1 — 
.’. Market ‘information was, nécessary ‘and thus provided’ for national 

and state occupational information coordinating mnittees 
4 ie “oes . : , 
,2:U.38. Congress, House Report of the, Committee on Education & | 
_. Labor ‘together with Additional Views, The Vocatyonal Edueatton 
-. and National Institute of Eduagtion Amendments of 1976, 94th : e; 
Cohgress, 2d Session, A976, H. Rept. No. 94+1085 to, accompany ~ og 
H.R. 12835, pp. 27-29. oN me oo ae < 


os 


way Ss en a. "9%, dias ee He SF eee & gh ag Pa els Pe daha Meee 
; iy _o Bas . . Se a a : 7 Mae a Oe Se one aad a se 
iets . : fo eS AES ale . 8 OVE a he roe 22 : D 
o on = 7 a = s a Came iad . 3 . rea ee eS Big. 28 a 
vo a! & : C : ea Pk ora bo sow i a : 
« *  ~fee’* 2 " ar a . a. a a : Oe a ‘y eee Ge Geo coe 
‘ r an 
. ” 


-(NOLCC’ ahd ,Sotcc’s) . “ pregéamp ‘vhich, purported’ to! impart: i 


"entry: level job. ski iis ‘would be évaluated annually..by aes a 
, data collected by samp Ting techniques’. .- ‘Complg. d.Jeaverg of 
: vocation 1. programs: would “thus ‘be followed-. and that. nformation 
would Be: fed: threugh thé.:-vocationa). edpcation. data ststem, (VEDS) « 
into “the staté and national occupational igformation cordnating 
committees for a bettef’ pidture of the demand for and. supply of 
workers Employérs of former. vocational eduéation students would 
be surveyed to get a qualitative assessmeht. of the: training:, being 
‘offered. It was also. assumed that a better planrfing process eee ae 
facilitate responsivehess to the labor market; consequently, stat 
boards of vocational education were required to work. with state 


for program and service offerings. #A systém of checks and, 
balances was introduced when Congress mandated ‘that the ‘plang, must : 
be: " (1) reviewéd by the state advisory councils. and others. for 
assurances, (2) “reviewed and approved by the U.S. Office of Educa~ , 
tion,“ (3} used as one ‘of the bases for evaluation by. the: Bureau. of. 
‘Occupational and Adult Education's Management - ‘Evaluation Review - 

. for Compliance. and Quality — ‘(MERC-Q), and (4) used as the basis of. 
the state boards' annuall program plans and accountability. reports. 

' The need. for. greater cooperation and coordination with other - 

' training ‘progr. S was alS0 viewed as necessary and was partially . 
adéressed. threudh coordination requirements placed gn the state | 

yoccupat ional Apformation ‘coordinating cemmittees, although Con- 
gress did not want to interfere in d¢cisions on who should offer. 

' “what courses. Congress addressed many issues “related to the . 

' ‘ability of vocatignal education to be résponsive to the -labor. 

market, but they did not fully address the need for flexibility. 


Thus, several questions remain: + is,it possible for an institution, . 


in this case education,. which takes" a great deal of lead time to 
‘develop and implement - programs, to show real responsiveness in the 
~ghort run? Are the expectations for vocational education realistic 
and how. will vocational . educators know when and if y are being 
. Successful? - , oO o , y™ 


This esate ‘shows. in a simple way the increased emphasis on 
' and interrelationship of evaluation, accountability, and piannihg.. 
Congress seems to have had ‘the bestginterests of vocational edu-. 
cation at heart when members prepared these comprehensive amend- 
_ments. ,they clearly felt that it was necessary not only to deli- 
neate gvaluation requirements but also to fry to ensuré that : 
evaluation data would: be nee for planning,: improvement and re- — 


ee direction. 6 a : 


¢ . ee “fe 
4 ‘ 7 2 “ 


6 Tener Datta, "Better Luck this Time: From Federal- 
Legislation Practice in Evaluating Vocational Education" 
in Handbook .@f. Agsevery Education Evaluation, ed. Theodore 
Abramson et al. ieeverty Hills, CA: Sage Publications, Inc., 
' ; forthcoming) . : 
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. . 
go! 


‘sand local advisory ‘couhcils and’ the public in ,developing. plans: a 


"_gréater ace par lity ‘and increased planning and’ cc 


“4 ‘yen 


a 


to be respqnsive to local needs, ‘there are’ those/who resist.man- a 
dated evaluations. and standardized cpa not because they do. / 


~ feeal-it would be inappropriate to compare programa within:a school, 


The full spectrum ‘of percep ions ; appears to be evident in vocar 3 


4 


; epectesed in — act. Ny 


‘The: fact: that Congress han ‘naridated fumerous - eval ations, - 

eration 

on not méan that programs have automatically. impraved -at: the 
pi cee LTevel.: Vacational:éducation is. part. of the ptblic, ea 
rprise.and this,the: responsibility, qt: ate. ife logal, govern~. ne 


ments, which: have. their. ¥poses, administrati 


atructures, “ eye, 
‘and legal and financial constraints. Since prog 


ms ‘atte’ designed . 


not want to be evaluated‘or held accguntable, but: because they. 
or from school .‘to school let alone fro state to state. Some are | 
‘reluctant to spend precious: prograin time gathering data that do — 
not. seem immediately usefyl simply to comply with requests. Some 
state boards feel-they lack the. authority to enforce local compli-. 
ance. Others have. né@ither the staff nor finances needed. to bring 
about the desired changes. . 

‘We.do not, in fact, know tie Sotial extent to which 
‘federally legislated evaluation requirements can bring about -: 
state and local adoptions of improved: programs and practices,’ 
nor are the motivations for state and, local compliance with: the — 
evaluation requirements in the Amendménts. of. 4976 altogether ee ar ee, 
clear. Walker contends that: ck ok oy a ae ao ts eo 4 


Like any ther sector of education, “state and decal 
vocational education administrators act on (or react 

to) legislated evaluatson requirements in terms of | 
their perceptions of the sanctions to be applied AE ea ‘ 
they don't comply, the utility to them if they do. ...-- | 
comply, and the level of resources (that must be... a 
expended intorder) to cemply. If those with the 

' decision.and authority prerogatives see the sanctions 
as high, the utility as high and the resources as low, © °‘ . 
it is very likely that compliance will be full and 
enthus?.!-ic. Similarly, if the kerceived pattern is . 
one: of low sanctions, lowyutility’ and highvresources, ¢ 
one would expect non- or only ritualistic eunerronee: 


tional education's response YEO the evaluation requirements | 


ry ‘ 
. * - =. ry - 
y. : ; ; ‘ :) 
4 2 te Y 
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7 Jerry P. Walker, He derni Evaluatton Ragutvenénie for ; 
Educattonal Programe: Conditions and Consequences, (Draft, 
Manuscript, Fort eonrenee Co., 1978). 
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4 “The primary purpose. at. aha eueicaeien requirements. as ome ae ae 
tended by Congress: is to ery: to give local. people and etate ~ - Oe 
administrators an dpportuntty. to ledrn how thetr programa gvt ev  *:. Jes 
‘s operating and to help. them improve ' thbee programe.® ° Comgresg 57 Se 
* does not (want, evaluations: that only pRoduce. na psonee- data’ that, yee Pe 
“ may .be useful’ to: thetn., and no a else. ” a a cra ae oa 
Evaluation, sadn, has, many, roles as ‘gpécified dn’ the ‘EBduca~ to ae a 
tion Amendments.of 1976. It: to be“used to improve programs ate ° 
the local level and to feed in fe) policy decisions at. the . national a 
level; it is to aid-in.making agencies at all ‘bevels more accoun- - nee 
table; it is to ensure that-certain important issues. are addressed. -° ; 
All of this in an atmqsphere where there is lack of. agreement )on. 
what constitutes. success in vocational education ane. even less . ee 
agregflent on ateeptable” levels of failure. pO ee, ee 8 2 


‘a ; ot ae ia i 7 . .° 


8 John F. Jeantnge, “Evaluating. Vocational Education: A Con~ .> cu ge 
#8 gressional Perspective" (Speech delivered at the National Con-" 

x ference on Outcome Measures for Vocational Education sponsored , 
_ "by the National Center for. Research in. Vocational Education 
and funded by the National Institute of paves in Louisville, . 
Kentucky, August meee J . ; cr 
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1976 came in response to. the need for more and better information. 


aes er eeaar er ‘and those with responsibilities related to 


10° 


CHAPTER. ~ oe Le ee 
| OVERVIEM of EVALUATION ncrivitiEs: | : pees ea" Pee Pa 
The evaluation provisiotis” ‘in. ‘the Education ‘Anaraneite: of —_ ow 


‘on the quality wand effectiveness of vocational education. programs. "foe 
Testimony before the Committee on Education and Labor prior | hi 2 ee 
to the enactment . of the: bill indicated that, improvements magt oe . eoa. 
come in the areas.of planning, account abglity, data ettetehe 

and use, and evaluation and foLLow-up.9” While this paper i 

primarily concerned with: evaluation, all. of these areas a : 
poker and ats -ConveGeed in a ae the } legiplation: aie ‘the 

‘pa © % a 


é 


This section focuses on the. groups with specific aye ieeton. 


evaluation. After months of analyzing doauments and making .« 
numerous - telephone *calls, this section was written tto provide 
a general,-overview: of evaluation. activities. ‘It should be .. - 2 a 
viewed rather like a snapshot taken’ with a camera_which ~ 
giveg the general picture, ‘but the detail is nof great and 

indy assuredly someone has been cut out of the scenes -Detail. 
‘was exchanged €or brevity, and those in need of more extensive . - eS 
‘information may find, in addition to other reports, the National’. 
Institute of Education's recent state of the art report10 and - ee 
the National Advisory Council on Vocational Education’ s annual : 
svarueen of state reports, useful. 11 a os _ 


yy 


-U. 8. Congress, House “Report of the - Committee on Education & 
Labor , together with Additional views, The Vocational Education * 
and National Institute of Education Amendments of 1976, 

94th Congress, 2d Session, 1976, H. Rept. No. “94-1085 to . 
accompany, H. R. 12835; p. 16+ - 4@ “ait Ww 


State.of the hee Review of eyatage ion Paoceduns. ene onae 

- Education, (Prepared by CRC Education’ and Huma ve lopment , : 
Inc.. £or;. the’ National Institute “Of: Education, * ovember, 1978) .. 
Overview, 1976 Annuat. Evaluation Reports, State agony. 

- Councils on Vocational Education, (Compiled by the Nation&l © 

, Advisory founcil on Vocational Education ,, ae boca ae 
D. Cap JAty 19770 : 


/ to conduct a survey of all available’ sources. An effort: was. 


‘limited time available». rand interviews were conducted. with 
» more than ninety. people in responsible positions, 1? 


- and- regulations (when. they expand rather.than simply. repeat) 


’ 


an overview of what* the groups were actually doing in thd fall 


behind: the legislation and the evaluation activity are then 


‘Seven: aig erent groups were igs. very. explicit. evaluation 
_résponsibillities ‘ah the; 197@ Amendments: * (1) the state boards, 
education, (2) the state. advisory ‘councils for ' 
-.wocational’ éducation, - (~) the Natiortal Advisory, Council for, 
*“Vocdtional Education, (4) the U. ‘S.. Office, of. -Educatipn' 7) Bares ao 
‘of Occupational: and Adult Education,. (5) the National Institute. 
of ‘Educa ion,. (6) ‘the. National Center for ducation ‘Statistics, - 
‘and (7). #he U. S.. Office of Education's Office of Evaluation :- ae 
and Dissemination€fqrmer ly referred tov as. ane Office of keene 
Budgeting, and hh al . | se 


- While represehtati'ves: of all the above groups. were. eon 4 oe: 
tacted for information, it was not the intention of this, praject .. 


made to contact-as many knowledgeable people as possible in the « 


-Vocational Education’ Act as, amended and related: rules 


are cited for each group's activities in order. to show, what, 
each group is charged with doing. “Assumptions / expectations 


discussed in order to better ynderstand the Congressional ‘intent 
and the perceived utility of e information produced through 
the evaluation. The part entitled approach/activities gives” 


of 1978. Perceived andeanticipated problems: are also included. 


The comments are those of the author based on information — 
from numerous sources. — . 


e , . 4 r4 


As stated in the introduction, sources were assured of 
confidentiality and are not identified in this report. 
The author appreciates: the candor with which issues 
‘were discusséd in an open dialogue manner and a sincere. 
effort has been made to present the information. re- 

‘ Bponesey: aw New ra 

| : 214 ~~ 
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vw islative bas 


oo ‘See : Nye: byea) “tn a the States. .oy pala Soca’ 
Sahat onal: agenetes and, o oo reqipiénte of; ‘unde . ON: aperating 
| the beat ‘posat ble. program of. poadédonat. adue¢ bionmny rs 
(A) each State shall, during’ the five-year pertod ofthe’ ‘state! 
plan, evaluate the. ffeativeness’ ‘ofeach program within: the. °° 
State ‘being Geateted with funde available under this Act; and 
the results of these evaluations’ shall-be usgd “to reviee the 
State's programa, and. shalt ‘be’ made readily vai lable. to. the. 


State advisory @oungtl and” + 


_ (B) each State shall evaluate, by uaing data. eollected, : whens. 
ever possible, ‘by stattetically yattd sampling techniques, : each.” 


eau ‘programy within: the. State whtch purports to tmpart entry ' ee 
level job skills according to ‘the extent to#whioh. program: eon 


pleters and leavers-- 
| (4) find emp Loyment in } Soaupartons related to" their. 
training, ‘and - 

(tt) are constdered by thec> employers to ‘be well reer 
ae and prepared’ for empgoyment, .* 
. except that in.na case ban pursutt of additional eduoation) or 
“training by program complétere or Leavers: be none tdered” ndga- | 
tively in theeé evaluations. J | rs | 
of ie 


“Regilations Implementing Sec. 112 (b) (1) (a): 


104. 402 oe ‘Fhese evaluations “ahatt held sf 
fad ‘Planning. and operattohal processes, such.as: 
£0) uality , and quar tabalery of. instructional 
- offerings;. = 
(2) Gutdance, counse)ing, and placendnt ‘and 
foltou-up: services; \ 
. 39 Capactty.. and condition of fakilities. eg 


-équtpment; 


(4). Employer participation in cooperative ‘programe wee 


of. vogattonal education; | - 
(5) -Teacher/puptl rattos; fre \ 
(6) reaenee qualifications. ae 


Regul ta. ‘of atudent achievement as measiired, for: 
example, by: | 
(1) Standard - oceupattonal. prov iatensi measures 
(2). Criterion: referenced; teets; and |. 
8) Other éxaminattone: of. atudenta!. skitle, ioe. 
ledge, ‘attitudes, and. neadtneae for, ental te 
emp Loyment. ssuccesefully. a8 
‘Results of atudent emp toyment Bucdess ae’ measured, 


ie 


ts 2 Yow eaanpia, te grok. | ee ee ee 
ae ae £ ob Rates gts employment: and ghemptogmett Sw 
ra (3): Duration of ‘emp Lbymént; qnd\) cae 

(4) Employer satiafaction with. eae es 

-  - yoeattonal edudation etudente | Yocompared with. 
(So penffrmanae: of’ adel ete ot had ge Aaee 
? 3 edugdtion.. es 


(a) The pesulte | of "additional seroisee as’ taueaned by: 
'. ithe suggested criteria under paragraphs (a), fb). es 
~ o,  . and (ae) of thie, acottion, that the State. provides. .. — 
"© > under the, Act to these. ereatay Seda ae ete 
(1) Women; ; is | 
(2) Members of atnowttu® groupe; 


(3) Haudtcapped!peraons; 
(4) Disadvantaged persona; and’ °*-. i 
(5) ‘Persone of . Limited Engligh- “speaking sin 


Egulations implemeiiting Sec. 12 (by (1). (B) : 


104; 404 + Spectal data on soupierase and leavera oe 

. (d) For the purposes of this section, 4 State 

“-' . shall report separately on program completers and. 

program . Leavers in adcordance with the survey 
instructions and sampling standards to be provided . 
«by the Nattonal- Center for Education Statistica, 
HEW , ‘ ; 
(fd, The evaluation data dn completers and leavers 
‘shall be collected at'a date to be specified by 

. the Nattonal Center for Education Stattettas, ABW,” 


ions. Congress in specifying the two 

evaluations listed above -- (1) systematic evaldations gf | all 
_ programs within a certain time, and (2) periadic reviews of :pro-- 

pbou to determine their effectiveness by following up. students 

. their success in ‘getting jobs--was responding to one of 

yd greatest needs and greatest failings..as perceived by wit~ ; 
nesses who testified during committee hearings held prior to the: 
enactment of the Education Amendments of 1976. . Congress wanted 
states to increase program follow-ups and planning: in order to 
becdme more responsive to labor market needs. fhe Vocational . 
Education Act before 1976 encouraged states to conduct these . 
evaluations and since those provisions seemed to have little, * 
effect, the new amendments a At. ; - 


4 


sa U. S. Congress:,. House Report of the Committee on Education. & 
“ Labor .together with Additional Views, The Vocational Educa- - 
- tion-and Natdonal . Instituge of Education Amendments of 1976, 
94th Congress, 2d Session, 1976, He Rept. No. a a085 to 
‘ accompany | H.R. 12835, p. 38. 
. _ | 
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Se eM re Results - from the ore tion: aye - “to. bd neea to goteeige 
en ae ‘programa, and. a summary: of then. ‘must-be 4ncluded'in the ual’ 4 
+4 °.accountability report. that iss mittedeto the U. §,.-@ffice of - 
“. “Eduéation. Evaluabion reports mugt also be made avaitablé to 

_ the state advi§ory, council on vogatjonal education in order a : 
— "for ‘them ae prépare| thefr own’ ‘annual evaluation report and int. 
,. order td better momiyor the stdteboard's.evaluations (see ~  . , 
_ pages 21-23: for mor eae on aa Soe Marina re a 
ye “and evaluation) . . a ee 


= 2 ae 

In poate) ‘Before a Geace wanccumetse held in: Marchg1978,_ 

i» sgme members of. Congress exptessed’ concern about the quality of 
~ evaluation data; :that. were -available.14 Ina period of. infla-— 

tion, youth unemployment. and taxpayer resistance, the bottom. line . 

‘on evaluations and the ability. to improve programs is whether 
they. can elicit information that can be used to justify - 
expenditures of federal’, aie aia ‘local Corhans. at a 


a ¢ 
mew, ae 


Other uses” can, Of- course, be made ‘of ingormakion from state’ 
boa¢d. evaluations of programs. Program improvement and planni 
are obvious uses of such information. Programs found *to have 
_préblems can be-helped to improve or be discomtinued. A 
strategy for ‘more effective in-service training for teachers 
can be developed when states know which areas need help. But” 
some people readily: point out that when an evaluation system is 
imposed from outside rather than arising out of felt needs’, 
initially the only use of evaluation is for” compliance, Only . 

_ over, time can the evaluations be adjusted. to lqcal- circumstances 
and put to some real use. Another individual. commented that oe 
evaluation data will be very useful to the if they eVer find. 
the time to do something with the information they obtained. 

.- These remarks highlight the néed fon evaluation to be dlearly 
« délineated and its purpose stated yas \ learly as- possible if. / 
maximum utilit is to be meade Of. F e information, present ‘ 


stivities.- The xesp nse ‘Of states to the. evalua~. 
tion-requiremefts has. béen varied. A number of states have ‘been’ 
using system c evaluation schemes for: years, some even mbre 
extensive than what the 1976 Amendments require: Others age. vy 
pleased that they have not been evaluating on a larger sca e° 
since they might now-have to change everything. Still others. 
are happy to have the federal requirements to encourage greater, 
compliance or better documentation. _ Sortie. are annoyed with the 


«& ae s 


Fag Uy S. ‘Congress, House Hearing Hetore a Subcommittee ‘of the 
Committee on Approprgations, partments of Labor and . 
Health, Education @#iW Welfare ‘Appropriations. for, 1979, |. 


gore CORSE? 2nd PERSSON) need, p. 340. y 
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oo pat Rewekuastiae abet cee eee ne hé evalpations, °°" 
. but fewquestton'whéether they mist’ dq thehy. they gt nérally onty .. { a 
', + .questign how they will, proceed ‘to\gain. the most: valu@ for the =f. 4, 
a ; @ffort, ; - ‘ id « . ; jie ae tid ce = 0 S89 . a “4 ee ve .. : oe : : 


cae 
are! 
a 


e ee io ght ; 


“Ve . we a : "4 dk 


oe 


“8 ~ "3 Most, states do.not have Wigislated’ state’ rSquirements, fpr" /2 s+ 3 


a y oe. " _ the’ evaluation of :vogati onal: education’. ' Consequently, the: or 
""% +: vregard ‘for whether those ee 


eee, 


is ee An accreditation model of evaluation is often used 
e _ .  - with a¥welf+evaluation followed by a team visit. rn re 


+ diclemma--some 
mean they c 


_ While many stat&s- feel that trying to measure student HS ee 
achievement is a gootidea, few are satisfied with the various — 4 
© Means available to do 30. Follow-up data are yathered through 
~ |’ S@hools, teachers, or sdudents-and range. from mall samples — ee 
rs, ik total populations. ah ee . - * * 2 oe tae! 

Ad . . | ™ ae cn ES 

| ; For extensive information on. how the states are responding 

to the evaluation requirements, see the report, "State of the © 
Art Review of Evaluation. Procedures in Vocational Education," 
prepared by CRC Education and Human Development, Inc. for the 

he: ee Education in 1978 (see footnote on page 


Problems. Most people in the states agree that the intent 
of the evaluation requirements is ood, but that too-mich is 
* being asked for too soon. ‘Séferal‘ States that had developed 
comprehensive evaluation sys$®ms prior to the ,1976 legislation. . 
_ agreed that a minimum of three to four years was necessary in 
* order to develop those systems and get ‘them to a point where 
‘they were beginning tg function well. Perhaps éven more time 
would be necessary’ in:-states that are initially responding to . — wi 
‘ ‘ : . : : . ow e. A ‘ 
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= ‘desine to” ‘conpily: | ide than a a. oe ‘tos ie ox So hecee 
-.: need.’ However,; the. legisl ation. requires that al programs must: 
reer 3 pe! evaluated. within five years. “A state :without™ systematic 
“evaluations must. ei¢her spend little: time on: development ib 
‘order -to distribute. the evaluation “Load: more, evenly . real the - 
alae eager initially | ands 
oe time period... tie - : 
Ke sources scarce enou 
forced are she. 


- former position. 5 
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CK frequen ly méntioned problen’ is the. Lack’ Of money . for-~ 
: Pott epta _ Some state people. mentioned they were having to - 
: ‘spend time trying to. con nce ‘people at the’ local ‘level’ that ~* 
evaluations will ‘bring s antial benefit™ to. them: wince. the; 
- money” for. those evaluations Will. ‘probably. come: out. of progr <e 
funds. ‘Evaluation. systems. not.’remain static, and. consider- =~ 
le energy. must. be ‘used in improving— and’ changing them Lf). 
rogram’ improvement is to. be, “a result.: Thus, “the ae) 
. oo once-:an.- evaluation—system: has - been’ developed. : 
~gtudent achievement, as: mentioned: previously,.. is. a prelen 
for many: states: because: Fria: Seape ea beta measures - are. 


. , = “ee ge oo ere 
states algo: ‘aeicia that little. cons deeekion Tse bags _ 


‘given. to. their particular circumstances For example ,. personne] °° oer 
_in some states feel. that process evaluationg may be more appropriate” meee 
“ and easier to do than product evaluations. States with large | mine 
- populations may have problems: doing follow-ups because ofsheer *. 

numbers, Gepending, on how program is defined.}. States. ,and, areas oe 

within ‘states with hedvy out-migrations haye /trouble tracing. 

students for follow=up ,. let signe eontactin employers ‘to - _ ; 

ascertain thefr satisfaction with former stfidents.: ‘In. addition, 

in those states with a-tradition of. ‘strong ‘local. control of 

education, state education agencies often have not had th . 

authority: to evaluate: logal, programs. . .Conseq sntly, they often 

conducted evaluations of: local education agengies .(LEA's) in _ 

- response’ to réquests from those LEA's, and. statewide systematic — 
. evaluations-were never developed, Now local and state agencies’. 
are. maven’: ai a new way: OF eink ng and oe ? 


There are other problems as well. ‘States wonder ‘how . 
ine cycles’ for evaluation in this legislation: c& .be 
hooked up, with the. seven-year cycles for their school ae- 

. creditation reviews’ in order'to reduce duplication. ‘Few states 
have ever collected \ary information:on the leavers of proarams 
and they . are not sur sot -how ,to ie or. yet fe aaah 


Fd 


oe evalua ions 
> ‘be improve ints.” . ; 
| st work gO. wedt 
ae 2a§ of the fntanstvetee 


re eee ’: take” ‘years. eee ahothee state: to. ome ip" t0. 
ake that” Set apt ar Spenser and get:.the” same= quality: ‘fF. 

_ ', follow-up ‘information. Confusion ana- ‘uncertainty about: ‘the... 
- objectives of. the-législation + th# 

|. compatible’ by: states=-long “rang [progran improveiont. veraue - 
eles immédiate need to be ‘ace 1 . e 
i" aiffteult for. them:to “know whi 
~@fforts. “Several :people™ “gonmenEed. “that 3 ‘more: lgagership. and: : 
CL coda, the. national: ‘level: ‘would: ‘be: helpful. ae = 


a 


= the: qaeston sti it. semaine ): what” a6 ‘state-boards: of 

“yecational education: need to..know:-and.do: ‘in, order: Grencia es, 
= progr att ‘improvement at thé local .level?:, Are: ‘they-getting.. ‘hag 

-information-from. the. mandated evaluations?. It-isi ‘beyond. the... :~ 

zvseOpe of this paper to. \qnswer- those -questions, “but.:such_ qies~"- — 

, tions should be consideréd-by each state. before the* next. ren Ne 
vision of federal: ‘vodational. education. legislation. is completed. 
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_oo a - ee EY aITEY Peach deate Lvisovy. ange shalt: Jaleo: ee ee 
os ‘evaluate, vocattonal education: programe, -gerbices and. sapietitee Pet aneeen 
ghey ¢ O88t8 ted, under . this hot, am pubttoh ‘and, distribute the! pasulis -: 

@ adi 


ve __thereofic™: (3)- “Fach St visory gounat! shall preparé.and. 
ve ~ submit: tothe. C mmtsstoner and #0: the Nattonal Agvtsory. Council 
ee aeioks under sextion 102, through the State bo@@Md, an annual 
oe evaluatton peports. accompanied by such additionde. comments of. ° 


“the State board.as the State board’ deems: appropriate, .wht h (A). 
we eugluates” the. effectiveness ‘of: the: vocational. educatton p. O~ 
at ee Bervipes , ‘and activities earrted out .in .the five year, 


A Bubnt teed under section: 107°: and seation 108, ina” “3 


7 2 By etalk oa Lee in.such progranes orotde 7. a a 

ae nd ugtiotetes —aamay - be: deem d necessary. ee ee ee, ao 

Se - + 4 ~ oo : ' z i ar 

re. ded 112 (B9(2) « ‘Bach stut Hormulating tie. piune tg? : 
: “putpe heap" pequtrements, shal ~Gnnugl Ly: consult with ther Sp % 

State advisory cduncil which- ahatl- ne the: State ,in- 


ee 


Pee “developing these.ptdns, ‘monttor the evalitions’ conducted by. 
“~“. the State, and fee the results’ .of these e luattons tn. i gall a 
as. ate es report feawerea. bys weoe ton 1056. OP cadige Bs "Bi, 


. 


\ , ste i‘ - . 0 ot > fea 


Rage. dooiemanting ¢ this a pestlany oa it eee 


ae Pies Wntotading programa to overcome ‘sex ee “Bervice 
_ we a amd adetivitiey, under the annual program plan, and’ 
Ve a, publish and distribute the results thereof; — 
ie 4 ee 8 (d) Asstet the State board in developing plans. for. 
“i State board evaluations under‘ the authority of  . 
. : _: See, 104, 401 and monttor' these evaluations; - 
ye . (e) Prepare and submit through the State board to 
a _ the Commiestoner. and to the. National Advisory , 
on - Counetl an annual évaluation report, accompanied: 
OF by any additional’ comments of the State board aa 
bhe State -board deems: a aa a oe ee! 


. 


_ Masuipt cis /on etations. ‘Beate 
ree 2 L “set, up un 
7 ay ydments of: P1968 to. ritay aoeg! Tay earcintpeeige in coe 
S planhing ana evaluation: of states’ vocational’ programs - , 
ee _. Generally; the advisory: ‘councils had‘onky Jimi ted: npact’ a canting 
cgi. On gtake programs, nd Congress. added: -varj ous amen endmevite a ae 
“la \dn .1Q76\ingan attempt -{o remeqy: problems ’ and.” to: expand. ithe. ok ce “a 
_ councrls!. Bunctions: © Congtess d finitely wanted thém’ invelved'. Sr ere 
‘tin ‘the enjire playning process’ and ‘in the :ménitozing: Brecesns. oe 
'. "In order to°dssure thatthe advisory ‘councils’ could ‘perform ° 5, 


.¢ 
. 


, independent evaluations ,. ‘Cohgress: ‘included: additional | amendments . Heng 
3" + \ in 1976: giving the’ councils control over -their .own- programming;,.’ wes res 
a ‘adminigtratipn,’ and- fysfis. It had been. dafficult for some. |. et. oe 
..- eouncils, ‘to bay negative things about state programs when ‘they: 7 
“, were’ depehdent.on. the state boards. for-their funds. 15°. The. Reg 
1 gs *. advisory councils. are generally’ expected o, $eé ‘that the: heeds: * soe 
aA - of the states' citizens axe being: met b¥@ gational: education’: - 4/3, 
and to see-that the state board does whatWE: is supposed'to, =.. + ,) 
. especially with regard to planning, evaluating and accounting ~~ 
' for..the’-distribution of fundg. |In the past, this function may ~ 
have -been of. ‘particular - impo tance in those: states whose: state 
|. ” .boards!were not separate entities. “One source’’mentjoned that. 
“ ., dn some states, thé administration of vocational education was: 
aon ‘sométim Peas a state departments of ‘education and. the ‘state - 
‘advisor uncits were “to help see thattertain aanhyieteabive 
_ 8 ‘tasks wene eae ignored. .Altkough ‘they have no’ powers ‘of 
ar “enforcement , councils! recommendations -are, meant. tae f 
_'¢ . ., seriously by-tHe states and are fameed. on to tHe Hebe § agen, 
. ceo) a Education and to the National. ‘Advisory, Council for Vocational | oe 
re ae aia annual . evaluation ieee ° eas ; oe py iy 


ne 
,a : _¢ 


Dg | Approach/astivities. As aight be expected): ‘state. advisory | nee © 
aes ,-equnc ave ‘responded -in. many different ways to ‘the ‘mandates. eee 
-. “Some do comprehensiv¥® statewide | ‘evaluations of somé very specific. 
« ..* things ,such as sex, equity. -Some evaluate the state HKoard'and “.,. 


their process’ for evaluation.“ Others ' ‘consider. how vocational ~ , 
we gy eduéation ce be Seve toned and amperes in their. ‘states. - Some, 


mye 


: 


t,” 


te. 7 2 
iy eae; : “UL 8.  Congeace. Mouge: "Report ° of the . Commike ee .6x) aGae Tent & 
ae A --{ Gabor together with Additional views, The cational, : 
‘ _ Education. and. Nationak AItstitute of, Educdtaan. Amendments." 
. oS (0-6-1976, 94th Congress:, 2d Séssion,°1976, H: ‘Rept: .No. 
m <i = 9491085, to accompany: H. R. -22835, Pp. oe me : 


Z 
: : wide : Oi . . Cr ra 6m yx 
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2 yo, . 7 ee . foe, we 
pet Wo abesbar ey” iat Me gs, ace ee ’ 


Bt ee eS ee eee a 2 . We - ; ee a % | ee b a Sa i - ue 
. ‘do evaluations that complement the’ state boards" evaluations while ns 
"+ -o@thers"do.spot che¢ks’.on them., “Many:.advisory council members *) 
eae go on state evaluation\visits dither as’team members or ohservers.- i) 
=" ‘Since some feel that tHey cannot ani should not do ‘technical, 9.0 0 1h)" 
i gaevalustions, /they spend \nare time holding ee “Re a 
hk subjective:-évaluations and/or working. with the stat legislatures. 

+  -Somé‘have contracted for third party ‘evaluations while others 
A. have done’ studies*similar-to those conducted by GAO. Most ae 
_* “focus their evaluation efforts on the total vocational .educa- 


.*. ., tion progeain thrusf in their state. -* 0 .. | a tae 8 
ae Pie a . wo >: 3 ; . is vu < 2, e.4 —— ‘ , , u* 
es Problems * Problems -seem to stem from-different. interpreta- - - ee 

: : ' a i eee oe * nied gh G8 2 6 pe eg ; 
oy: . tions of words like monitor and evaluate. Mast’ state advisory ely 

"..., ° eouncils have good. working relationships with their state °°: 3 


fo % boards. However,’ since both. agencies. have legislated responsi- ©: - 
"2. bilities for evaluation; conflicts over the charge.sometimes - an oe 
-do occur. . Most councils function primarily in an ‘advi'sory Dy dog ae 
.. °.* Gapacity,. but many prefer ‘taking active roles in establishing 
ty state’ policies. Some councils sée themselvés clearly: as mde 
.. +,,, advocatés while others see themselves imarily as adversaries +, ~~ 
7. ot: largely because of their advisory an@#evaluation.roles. The 


— "’ two roles are sometimes conflicting,“but some centend that they ie, 
it could not serve in a proper advisory. or advocacy capacity if © os 


- . they were: not able to evaluate or assess the ovérall effective- oe 8 
+ «ness of.vocational education in their statés and thus know | | a 
— what“to advise. ‘A few are riot clear about their proper’ roles, . oe 
“and others are attempting to clarify these roles and achieve a ee 
7. working balance. | ° mat! 
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“ational Advisory council 9 


woe. WW legislative basis. 8 00 poe 
 Sea,_162 (b) The: Nattonal Advisory Counoil shall’... oe og ee 
. a) neveaw the. administration: and operation: of vocattonat we 
| education Snogeanp under thie apt, pnd other pertinent Laws 8 eS 
. affeeting vocattonal education and. hanpower training (inoluding... 0° ')": 
the effectiveness .of auch programs th-~méating the purposes for 008s 
- ‘whigh, they are established and operated), make recommendatione. 0 


with respect thereto, and make annual reports of ite findings . | 
_and reeommendatibna (inaluding teoommendations. for changes..in i Pog oe 
the. provtatons of this Act and .euch' other pertinent lawe) to the — eres 
Preatdent, ‘Congrest, ‘Secretary, ‘and Commbeptone?® oo ee tue: eae 
a ; (6) .conduct Sdepaniend coatudeiane. of pnoprane earrted 0 08 nse 
Ee out under thts Act and publish and distribute the requlte . 8 ey 


* "It is sometimes difficult for-advisory councils to ascertain | . 
their own effectiveness and tf know where to channel their - = ... 
talents. Several ‘meritioned: that they could use guidance on mow. ty 
to more, effectively use the.information:they find in their = - tee 

- evaluations. ° ‘ + a ee —.* |e? 
' : é , ii a . sc , ; . : s q. so : i 
- “Comments. While state ladvisory .councils perform worthwhile © 
functions, expectatidns may be rather. high for what they can | : 
’ yealistically ; The work some do as coordinating councilg may ~ 
be just, as impgrtant as what, they do in evaluating and advising. an 
Many. Géuncils feel that ney lack the gi ee authority. and . 
finances to conduct: evaluations in. the.t chnical sense & the. 
word and hope that their nterpretation-of their mandated 
_respongibilities Oe ee ne In many cases they perform 
assessments rather; than evaluations.’ Councilg want-to do what | 
- is best for vocational education and “students in their states, °. 
but what is. best is not always ae or unanimous. - In addition; - 
ée 


= turnover and commitments tg oth full-time jobs will always 
‘Limit’ the amount ears and en voted by 


gy that, can, be d 
members to council activities. - a : 


; — “tye : 
‘- [he councils do seem well-suited for asse 
their state boards have carried out the plans s 

. five year and annual plans. The biggest issue regard. to >. 

. council evaluations is that there no provisioh for resolving ; wo 
differences when their.findings are different from those of the \ 
ptate board. Not that ‘one is necessarily right and the other 
wrong--but surely there is gomething to be learned from the 
discussion ‘and resolution, if possible, of those differences. 
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| * | asgumptions/expd®tations, In. the House ‘report! ‘that: accom-*. wa 
. panied the Education -Amendttlents of 1976, the House ‘Committee (o) | 


EdueAtioh and Labor: commended’ the National: Advisory Counéil on)’ 


/ * Vocational Eduéation and said that it had made itself into one. os 
'.'of th@ Sutstanding. advisory councils in the: federal government Jl6: "°° os 
‘They also; however, expressed concern. about, the way. some.duties ) "oe. 


had been, performed particularly those related to evaluation. f° 2”. 
Even though the National Council hada clear responsibility J. 


‘under ;ffie law to evaluate the performance of the Office of Educa~— -_ 
nc ~ 


tion rrying out its administration of the Vocational Educa-" | 
tion Act, the committee found that it had not been as-effective: 


as they had desired. “By simply: mentioning this in the committee ~~ 


report and by not changing that requirement, Congress let it be... 


known that they thought this was an important and. appropriate 
function that should indeed bé carried out. The other provision, | - 
(6) above, continued to grant them the authority to carry out any 


“kind of evaluation that would be ‘necessary to offer well-founded . 


: rid 
v 
fe 


advice to the President, Congress and others. Congress appears 
to want objective lay opinion on ‘the. nationwide thrust of/{ voca- 
tional education from people who are!concerned about, but in-- ° 


dependent of vested interests in ‘vocational education. - 


Approach/activities. As part of the National Councils’ 


: SS es so 8 ee ‘ 
responsibitity- for. reviewing the administration and operation of 


vocational egcation programs, they began a two-year study of the = 
Bureau of Occupational and Adult’ Education (BOAE) in September. - 
1976. Phase I of the gtudy was ‘completed in, October 1997 and an 
interim report entities « Study of the Admintstratton and ,Operda- 
tion of Voeattonal-Teehnical Education Programe was published in 
December 1977 by the NACVE.Task Force on Administration and . 
Operation of Vocational-Technical Education in the, U.S. Office | 
of Education. During Phase I, tas force members focused’their | 
attention on the Division of Vocational and Teqhnical .Eddcation 
(DVTE - the group that has primary responsibility for° the | 
administration of. basic grants to the states) within the Bureau. 
after conducting numerous interviews with state directors of a 
vocational edugation, professional staff members of the Division ae 


- and the Bureau, and others, the information from.all the inter- 


views and reviews of documents gvas analyzed and synthesized into 
the findings published in the interim —% In‘that report, 


Ca] 
— U.S. Congress, House Report. of. the Committee on Education & - 
Labor together with Additional Views, The Vocational. Educa-. 
tion and National Institute of Education. Amendments of — 


4976, 94th Congress, 2d Session, 4976, H. Rept. No. : 
94+1085 to accompany. H.R. 12$35, p.' 31... . » * 
: hae ee | | 
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administration of programs and aubsequently state operation of - °° | 

| em’ for the: communi-  e 

): cation, coordination and follow-up of assignments, and the Office». 
of Education's jack of a clear.stance regarding vocational =v... 
_ wogking on Phase’ II’ which is focusing on the degree and quality fg 


“programs ,were identified: “®uch. things’ 'as 
for program. specialists, an inadequatéd sys’ 


of programmatic and. functional services being made available to _- 


the states by the Bureau. A final ‘report.will be published when”. ©. 


the second phase is completed in.early 1979 and should ‘include . 
‘conclusions: and: recommendations from both phkses,” - he = a e 


education programs, e National Ad isory Council is also..— 
authorized to conduct)studies, hearings and independent. evalua= 
tions as necessary. While the council does not usually conduct 


"rn addition to the review of spre ener of wocktional.. ., = 


} 
ray 


independent evaluations per se, mémbers have participated. in the. oy 


Bureau's Management Evaluation Reviews for Compliance ‘and — \ 
Quality (MBRC-Q, see p. 28) as participating team members. on site .. 
visits to indiyidual states; conducted hearings on vocational ...- 
education for American Indians; and: contracted sien on 
the impact of CETA on- institutional vocational education \among 
numerous other activities. The National Council is more inclined. 
to.1se- evaluation as.a tool to gather data/information that’can | ¢ 
be used in providin§ advice to the President, Congress and others . 
rather than as sémething that can be used to help Bring ahout 
compliance or goo directly improve, programs at the local level. 
- Consequently, while NACVE could conceivably conduct evajuations 
- on any number of things that relate to the national thrust of 
vocational education, specific evaluations will probably be done  - 
only if they are dictated by the subject matter agreed’ upon in 
“their annual program of work. . 


= 


‘ 


Problems. Like any advisory group that must play. the roles. — 
of both advocate and advefsary, the National Advisory Council. _ 
for Vocational Educatjon is faced with difficult evaluation aa 
judgments. In their efforts to improve the vocational education © 


g system, they must be. concerned with how the information, reported 


in their evaluations. is used. Information can be “uged by those 
who oppose vocational education as well as by those who want 
to improve it. Consequently, all potential statements must be 
weighed carefully by the Council as to their ultimate impact 
on, and improvement of, vocational education. Some critics 
contend that the National Council. has not taken sufficiently 
_ strong positions or filly reported all controversial findings: © 
The Council must be concerned with long run benefits. and with 
providing a basis for improvement. : me 


oe: studies that. might. provide inf 


little impact, if nothing could@ in fact, be done about it.. 


ry Pa 5 é, 
vy 


“Bemources lage ‘a: ‘problem togean 
* ‘they have. not, been able.to: und ae 


th xe ineretore ce 
| tional. trends that. Se 
am vous be useful in their advis pryreapacitye ae oe ty oe 
a4 ee 


Comments . It seems “moet: appropriate for. the: National 
Advisory -Council to keep track of the general | ‘féderal. ‘Level 
administration of vocatiofial education programs funded: under 
the Vocational Educatjon Act. Because its ‘member's, are concerned, — 
one should.expect them to demand the best of vocational. education 


even if it means being criti¢al. - No one’ should expect miracles: 
fron. such valuation activities and their resultant recpinméenda-. 
_tions.- 3 the evaluation process ig to work-well, the organiza- 


tion being evalwated ‘must be both receptive to chanye as well: 
as capable of changing those things that would regu + in improved 
administration and operation. ‘For example, if NA were to find | 
that the Bureau of Occupational and Adult Education were grossly .. 
understaffed in-a particular division, a recommendation that. 
additional positions should be ‘allocated afid filled. would. have. 


Providing: thorough documentation and realistic recommendations 

on. the federal level administration of vocational. education | 
programs’ could: serve to keep Congress informed about various - 
developments and offset future criticisms. The National’ Advisory ( 
Coyncil is hot equipped to monitor the day to day activities in. . 
the administration of votational -education and should‘ not, be 
expected to.do so. ars ms. 


~ 


m 
Lets doubtful that the navionals eaeiseey éoureit would ever 
get involved in evaluatigns of programs at the local level.. 
Policy research and informal evaluations‘ that provide. earl 
insight into problems long before key decisions-making points y- 
are reached are two essential evaluation-related areas in, which 
‘they could make peagnt secant contributions. 
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| “yy: Ldgislative Basis. © Se) i Se er. 
a Sée. 112,. (a) In order for ‘the Federal goverhment ‘to eo ee 
| ‘aestat the States in operating the best posstble programs of 22°. f lL), 
v8cational education. .\. a ore a ee ee ae 
(2) the Bureau of Oeownational and Adutt. Education’ halt, DS ee 
in at least ten Statea a fiscal year during the. pértod beginning \\ wu! 
J. 4s: Oetober 1, 1977, .and ending. Séptembhéer, 30, 1982, gonduct. a nevtew”.™ pape 
-} analysing the. strengths and weakneases ofthe programs aseteted... :* 
 ".. wtth funds available under thie Aat within those. Statea; and the... 
na Deparenent of Health Bducation,:.and. Welfare shall, in ‘the game.” 
be pertods *conduct fiscal audita of such programs within those 
¢ States. , a ae oe a ee 


% 


y7’*. °“Agsumptions/Expectations. The Management Evaluation Reviews. Soe 
_. for Compliance (MERC's). conducted by Bureau of Occupational:and oi. 
Adult Education, (BOAE) staff are designed to.assess the.degree > ia, 

of state compliance with the Vocational Education Act and its | : , 
related regulations as well as other federal legislation such as: 

° (a) the Adult Education Act, (b) certain provisions of Section 112. 
of the Comprehensive Employment and Training Act (CETA), (c) Title 

IX of the Education Amendments of 1972, (d) Section 504 of the 
Rehabilitation Act of*-1973, and (e) Title VI.of the Civil Rights ¢§ 

Act of. 1964:as they relate to the above legislation. These a 

- state-by-state evaluations were begun in March 1976 in direct 
‘response to the General Accounting Office's 1974 report that 
criticized the Bureau for not adequately, monitoring the flow of). 
federal funds into vocational education. . a 


The Education Amenditents of 1976 added the provision that 
the Bureau of Oceupational and Adult Miucation must ‘also analyze. 
the strengths and weaknesses of the state, vocational education 
programs being reviewed. The amendments”also require the Depart~ 
ment of Health Education and Welfare to conduct, a comp ehensive — 
-fiscal audit’of each state's program during. the same p riod. : 


' oo a 


Bad » 


oe NS 


“ i7 y.s., General’ Accounting Office, Comptroller General, What ts 
the Role of Federal Aasistance for Vocattonal Edueatton?, 
Report to,the Congress (Washington, D.c.: U.S. General 


Accountitg Office, 1974). 
A 3 ; . 


A 


pyaar to 1996, the Office of eee aceanieA. a  guéat dean 7" : 
oan e parsrveas ‘ahd. detailed data. from the: states - ‘butthere’ ‘was 
“no follow-up on this. paper compliance. — “Vogational : ‘education ¢ ee 
"was handled: like a revenue sharing” program’ ‘and there was little. el & par 
effort to assist states or to try. to: correct percefVed misuseg 9 @ \i" 
of federal money. » According to the Report fom the Committee on 00 0" 
“Education and ‘Labor, Office of Edycation officials often did not 
‘feel authorized to go into a state to ask: how federal funds - 
were being spent.. Consequently ,. the Committee t' compelled . ; 
to include the above provision. . The report stated, In proposing - 
these amendments, the, Commdttee wants to make ch an that it does 
not bekieve that the states ane squandering these funds or that. 
there ane widespread illegalities negarding thein use. ALL = 
we.ane trying. to do is to assure that the best possible use Ab bedng oe 
made of these funds; and we bekiewe that effective oversight | Lt a 
of these expenditures, dneLuding mongtonring, a eee and Se. | 
auditing, are neecaeany to assure thas result. 18 . 


Congress is interested, in the ingormatiion that is. found ‘in Lh, 
' these reyiews, but members are also concerned with providing a. - 
‘mechanism that, reviews states" compliance with the, legislation 
and holds them more accountable for the expenditures ‘of federal : 
funds. Members of Congress also showed concern about the need 

to evaluate and improve programs when they included the Peceion 
on reviewing | strengths and weeknegses: ; : 


Information. from thé MERC review’ is distributed’ to several 
parties. Some information goes to Congress. in the Commissioner . 
of Education's annual report on ‘vocational education. Thedata 
are compared and a few of the most persistent problems in com-~ 
pliance ‘are pointed out. Trends. in areas of. strength and weak 
ness are’ also included. An executive summary for each state ~ 
reviewed is also prepared and submitted to the Office of Civil . 
- Rights, the National Advisory Council for Vocational Education, © 
and the Department ria and Welfare's audit y 


ency. , 
pee a 


The process of going through a review and the information S gee Oh 
that results from it are of greatest use to the Bureau, and to. 
the state board of vocational education in the state: being 
reviewed. Reviews may have the follewing consequentés: 


‘ . g 
re Fy 


® 
U. S. congress, iiouse Report of the, Contnit tee on Education &, 
‘Labor; together with Additional Views, The Vocational 
‘ Education atd Nationk& Institute of Education Amendments oe 
0f -1976, 94th Congress, 2d SeSsion, 1976, H. Rept. No. , ” KO 
94-1085 to accompany H. R. 12835, pp. 17, 28, 29. e 


18 


¢ 


4 


-. States ‘usually have .4 ‘better. interpretation. ‘of ‘the: rule: 
and. meguiacions: after going: through | a tne : 


States usually, achieve. improved documentation, a 


; and follow-up. 


-* Probem @ areas are. Chere etre Reams : “to. pen” ce Poa 
eee Ey and cont icm problems ‘for the’ states. eo a 


dipr 2ig. 16e 


ni 


States learn from their self-evaluation by Looking, 


at: themselves. | re ae ee = @ _ Fise. 


BOAE}gets a ‘better. amaaretandine ae ‘eeonieni: facing. 
the states, particularly with regard to (a) ‘the‘variety ” 
| o£ iriterpretations that can be made of the rules.and © | 

(b) the state. structures into which those. ee 


tions must ae i a . an 


BOAE ‘Veatne in what areas states “yaed technical paeigson 


Approach/Activities. The reviews that the maieca doce! 


(of at least ten states each year) are of compliance based on. 
documentation.. They go to the state scheduled for review and. 


‘give the 


that are 


back and 


people. 


‘staff, a 
‘National 
‘sible for overseeing the particular st@te, and re 


state board of vocational education ‘the regulations. 

to. be covered. ‘Six weeks later the MERC team comes 

spends a week. checking the documentation -and interviewing 

Typically teams have twelve members and include MERC. . 

representative from Adult Education and one from CETA, a 

Advisory Council member, a Bureau. staff member respon- + 
oe 


from vocational education and adult ‘education in a prale that — 


is scheduled for a review in the near future. ~~ c+ 


In conducting ¢ a review, team members assess -the extent to. 


. which the state is following. its approved state plans, and the 
degree to which the procedures being followed. are ‘consistent 


with each relevant regulation. For example, four judgements 


could be made on compliance with any regulation: (1) in*com-_ 
pliance, (2) not. jn compliancg, (3) to be determined, and (4) 
~~ not applicable. To be deteramcned's are given when there are . 


changes in the rules and regulations~and the state has not yet 


had time 


to change and adjust its proc ures. When a state is 


judged not in compliance or its compliance is yet to be deter- 
mined, team members make regommendations and assign follow-up 
responsibilities to appropriate Bureau staff.. Theoretically, © 
drastic pee a eg could result in the withholding ’ of = 
federal funds.. A "Q" for quality was added to MERC when - the 


i a eee: 
a, he oe 


‘ ’ ,, Paks " . ‘ 7 
3 ie - i . Co e 2 
on : = ve: | 1 a 
poe a ae ooaeee po rae a ee ete Oe a” or a a 
"Bureau was told to’ review the strengths and weaknesses Of voca= * 


-. tional education. programs. Since Janyary':1978, in vaddition to: —. %. eee 
- reviewing compliance with the regulations,, the teams: have been. 
-- Yeviewing quality of (1) state vocational edudation'planning pro- | 
_» cedures, and. (2) the procedures for the follaqw-up of completers - 

. ion | Modules ‘for review~.- 
bilfty of voca-_ 


‘Ning ‘the states' systems for evaluating the a bi: 
- . tional education programs are’ being developegfgmr should be pilot: » 
tested and in use in the Spring of 1978.” A couple of -other modules 
to be used. in reviewing the quality of other state.procedures will - |... 
probably be developed sometime in fiscal-year 1979. Asiwith |. 
‘the compliance section, the quality. review’ starts with ddocumenta-.: 
‘tion. There is more dialogue and interviewing? though, and’ team” * /. 
.members usually try to review the rough-draft of their final © = 
~. report with appropriate state people to make sure’ that they have . 
perceived the state situation and environment correctlys - Strengths 
are identified and recommendations on weaknesses are made in the «=~ - 
‘report, but*states do not have to accept any of the recommenda~" 
tions. . _ ee > ee 
. At the end of fiscal.year 1977, the first twenty-three .. 
reviews showed the states were 90 percent in compliance. In fiscal 
year 1978, the ratd was down to 80 percent. There were many 
more to be determined's because of the changes in the regula- - 
tions and the lack of time for a ee their procedures... 


, Problems.‘ The problems for MERC-Q@ dre the same ones that 
",. anything caught between two administrative systems would face: 
_ actions require long, slow processes, paperwork multiples, and. 
issues tend to complicate themselves. States have been coopera-.. 
tive and understand the purpose of the review. Whether all states 
accept the *purposes of MERC-Q as a legitimate process that brings . 
. about improvement is not yet known. . Follow-up and technical — 
lena assistance may be more difficult and less personal now that the - A 
_ regional office personnel have been centralized in Washington, 
« D.C. 3 , 7 * . 


Comment. The Managemént Evaluation Review for Compliance 

an@ Quality appears to be working fairly well. States seem to 
ha¥Ye accepted it-as a given. Even though state people may feel . 
that the reviews are often too detailed, they also comment that « 

they have been useful. Earlier sharing with all states, infor- 
- mation about the regulations to be reviewed and their changing 

‘interpretations would be helpful. & 2, *& — 


. 4 * * 
ie a / 
, 4 : 


pete ee Pe i wet : 5 ie roth ag he oT ERTS! Ae rye ae rete 
. * fs ‘ . + . 
1 ae a eee ‘ oe ‘ 


Te. is ‘aigfioult to. aad ‘vhather: ‘cheae. Rinae: a of oariphigndes ts 
 gevilvs: ‘actually help to improve procrama. Sevier apeseonnen haere 
“4: may 8erve as a. feedback. mechanism to Congress if it turn s-out - 
_ that -some items ‘or approaches are not as eff tive ore. ficient 
“in practice as originally hoped. / It should, bé’ noted. that cone 
eee ‘on ‘paper ° does. not necessarily mean . compliance in, prackicg,”, 
No state hag ever last its federal funding because of a:poor | 


. showing ‘in a.MERC-Q.' Therefore, a question is ‘raiged about. ‘the. 


‘incentives. for states to do anything more than. comply OR 
paper. : Continued dialogue between the federal and state level 
‘and subsequent technical assistance is needed if neghingei 

. improvement in vocational. education is to: result. 


. ‘andlprésence af: National Advisory ‘council members , on the... 
ray iew teams has given them & unique. opportunity to observe, ~ 
4‘ssess, and make: recommendatiéns on the Bureau's procedures, — 
us facilitating their quest to review and improve the: adminis- 
ation and operation of vocational education programs at the. 
‘national level. Since the General Accounting Office helped . 
bring the Management Evaluation Review for Compliance and 
Quality into existence, it. would be useful to obtain their 


ae 
comments on the MERC-O. process prior ‘to ‘the: next round of me 
vocational educaeson neghoearaer in 1982. fo) a ee a 


en ea. ee eee ae 


Le islative: mai 


eae, $23" (b) (2) In addition i ‘the other. duthontties. ; 
vesponatbilities, and:duttese conferred, upoy ‘the Nattonal *.. 7. © 
‘Inatityte of Educatton (hereinafter in..thia. eectton. referred foie ie 
as. the "Inabitute") by section 405 gf. tha General Education Pro-. 
_. -otetone Act, as amended by this.Aat, the Institute ahall under... ee 
-, takea thorough evaluation and tudy’ of vocational education |. 
programs, tneluding such peo gnine conducted by. the States, and. 
-. gueh programs conducted. under the Vocattonal Educatton Act of . 
¥ $1963, and other related programs conducted under-the Comprehen-~ 
s™-- - gtve Employment and Training Aat' of 1973:and by .the State Post- -"' 
Secondary Commisetons authorized by. the Education. emer dneny: of 
1972, Such ie study shatl. tne tude | ws 


. . (A) a atudy of: the Wistui bition of. ioaattendl edusatton 
. funds in terms of services, occupations, target populas 
tions, enrollments, and educational and- governmental levels 
"an hat such distribution should: be:in order to meet the 
ee eateat human resource needa for the next 10. egress 
; ; 
(B) an  eeantnalion of how to achteve apusiianee with, and 
enforceme of, the oe of. applicable vaue of the . 


United Sthtes; , 
x ; o, € — 
. tok (C) an analysis ag the: means of assessing program quality 
and effectiveness; —- . . 
: ; -%- 
( . 5 
Notes. (D) depending on the level of Pree apaatats tio the -% 


Institute, not more than thrée experimental studies to be. 
administered by the Institute, in cases where .the Inatitute | 
determines that such. experimental programs’ are, necessary to 
‘earry out the purpose of clauses (A) through (C) and the 
, Commissioner of Educatton and he, Secretary of Labqr are 
a. authorized, notwithstanding any proviston of any other lau, 
- at the request of the Institute, to approve the' use of. 
grants whteh educattonal or other agenctes are eligible to 
recetve under such Acts (in cases where, such agencies agree 
‘to the uses of such Grane), ‘in order to carry out sch 
paper enone a! programe; 7 = = _— 


(E) ‘a review and ‘evaluation of the oP ane iiaiees of pro» 
Hage 8 . grams funded under subpart 5 of part A of. the Vocattonal 
coe te Bdueatten-Act of 1963 (Consumer and Homemaking Education), 
a - "and to make recommendations for the redirectton and the © 
. improvement of programs at all levels punsed under- such 
1 oe subpart. : 
, - 


_Assum tions | Ex ectations.- ' Garin Wi: Radclitte, ‘the. , Minority ; a s e 


Counsel. to the U.S. House o Representatives" Committee ‘on: Edu~-.. 
cation and Labor, said’in.a presentation in 1977 that ¢he’ NIE 
“atudy tea da ‘one-shot (ef fort) for Congressional | purposes, to. get, 19 
“an independent evaluation of where voeattonal education ts naw. 

It is clear that the study was intendéd:to: provide Congress. with 


objective information to serve as part.of the basis for future. 


e 


gf 


legislation-~particularly i in ‘1982 when fhe. present Vocational ° 


0. eevee Act expires. eh, ahana i‘ ee 


bong 


(2) an examination of howto aes sae ae with “applicable 
-laws of the Unitéd ‘States, (3), an analysis of the means of §.. -” 
assessing program.quality and effectiveness, and (4) review: and 
evaluation of the effectiveness of consumer and homemaking’ . . 
education programs. The National. Institute of Education prepared 


a plan of study that was sent. to Congress.-in December 1977 in’ 


- which the plans for. these four parts and the specific research | 


“ERIC” 


. projects were outlines. In addition to the specific. studies, = 
the plan Buareo that. the undertaking also would peeUhe * ‘um ‘two 
products: rs aan ' 


« 


~ © of deceription of the vocationat ‘eduzation a 
enterprtse nationally --a "fact book" on’ © 1 
vogattonal education -+ designed to illuminate 
sharpiy the Federal role in tt; and 


A systematic account ‘of Lie changes in, we es 
the voeattonal education enterprise ea | 
that 2an bé athributed to .the Educatton 
Amendments Act 


. 


J 


19 John F. Jennings and Charles W. Radcliffe, Commentary on.. hegie- . 


eatich and Development, 
The Center for- 


latton Affecting Vocattonal Education R 
Occasional: Paper No. 27 (Columbus, Ohio: 
vocational EOuesEeOn gd) eRe aee 
_¥ 

20 U.S, Department of Health, Education and Welfare, National. 
Institute of Education, 4 Plan for the Study of. voeattonal 
Edueation,. Transmitted to the Congress, December\30, 1977, MN 
p. Vv. . a os 
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“Ag stated previoubly, Congress. was disappointed with the) 2.0), 
-” wational information on: vocatiofal education: available for their °° 
‘use when developing. the 1976 Education Amendmenté. It is dif) +). 


- ficult“to ‘say whether this part of the NIE study. is being dove yoko 
--- with an eye towards being abje to take implementation obstacles . st. 
4 into: consideration when prepating the 1982 Amendments or whether... Joo 
it is.a way of helping ensure better implementation. and Comp SE eat 
‘ance since the very .existence of a Study.may stimulate change, “?. 0. 
®, In any.event, Congress felt a-great enough need for a 
thorough evaluation and study of vocational education to provide 
$4 Filton coe a: congress: cual ly alanasted Study. It appears =: . 3 
~ [fkely, that NIE was chosen to. conduct the study because they ° = #\) oo". 
gould: respond directly ta Congress‘ Concerns; they were knowe"" 7 0" \. ow 
ledgeable about education, evaluation and research; and they would... 
be less subjected to influences by the executive. branch and —. ra 
others than other possible organizations. Therefore, their find- 
ings -would be more likely to be.objective and. reliable. ©  ~" 


It is unteresting to note that Congress earmarked a specific, -- 
__percentage.of authorized funds for the home economics evaluation. 
The Education and -Labor Committee commented that,. In the: past,. . 
vocationak education research, exemplary and demonstration 
-, projects, curriculum, national and state advisory councils have 
~ ark too aften ignored home economics programs asa pant of vo- 
cational education.2! It was their intention that consumer and 
homemaking education programs: would profit from the findings-of 


their mandated study. “a . 

eon ee ee oh ne ai a 
The committee was also very emphatic about the dates on 
_' which reports should be submitted £6 Congress. They noted th 
too many studies were conducted that produced information too. 


late to be of any use to Congress. ; e. 
ie . : a 
' While the National Institute's study is being undertaken ta 
, fulfill Congressiondl needs,and should-play a major role in the 
reformation of federal Vocational education policy and legisla- 
. - tion, its findings: may also be useful at other levels. At least | 
there are many people -inathe vocational education and related 


communities who have high expectations for this study. ° 


~ 21 U,S. Congress, House Committee Report on Education and Labor » 
p. 61. . 


pesanke ne NIE, aul 7 ee err een 


ae an. overview: of: their aetigs ees 


“ dttes. Por more details by: ane ‘their Plan-for- the Study'of Voeas ?. * 


tHonals Education £qotnoted: above: on page’ 34. - The: ‘pngoing. and 


Paonia studies’ are: jisted. under the. ‘four piers coda ae 
" Congress. ae oo ae 


ae, : a a a 


pe: " Dist¥ibution oft Vocational Education Ree 7 
Ae - Descriptive Study of the ‘Distribution ‘of Feder ie 
“State and. liecal Nogationat. Pancsion: Funds, ot 


“This study’ will, ‘examine the ways funds are oe 


tributed among and-tsed by vocational. programs, | 
and the extent to ‘which thase expenditures’ Gonz, 
tribute to realizing. the ‘purposes Of federal - 
_policy.. my . a sre a Cope” ee 
: - a re 
. Contractor: University. of" ‘California, Berkeley, 
Time Frame: : , September 1978 + August: 1981 5 


Be “Meeting the special. Needs of Special Groups: 


Using information from other studies such’ ae ‘eh 
one above plus doing community case’ studies, the. |. 
project will assess, the. degree, to which .the needs 
of: special populations: are being met. 


Contractor: In the process of being ehesene 

Time Frame: January 1979 - . October 1980 ae 
C; - Projecting Human ‘Resource’ Needs and Funding: Levels a 

and Distributions. | - to* am, A 7 


This project will ip etaipe ee look at vocational || 
‘education in the larger context . of human resources, 
and economics. 
~~. 
D. Simulation Model , 


Pe deat 
’ . 
e 


_@¢ 


In order to simulate the effects of changes’in the, 
distribution of vocational. education funds at the 
federal, -state and-local levels, a prototype. model 
will be designed and tested. 
’ " ; 
Both C. and D are in the early planning stages.” Large’ 
oes of the work wi eee be done. within NIE. 


See 


“ghe Retdete of vocktional: Baication: on Participants. ee 


Sea hy riviewing previous: studies: ‘and ‘peanalyzing ° os : 
hha  . vardous" data banks, - this. study; wil ‘attempt: to — 
eo i. ee - determine the economic and: non-economic“outcomes: ee 
RM dee of. vecansonee Moan are for: tbe peneicl:,; ae 
eee os oe i. ye. eS EE Bae nt 
mt Oe 2 eS = ‘Contractor: “tn the procdas, of. boas selected ae 
ee a on “et Time, Frame: «March 1979 ; =" to be. determined . ee 


Iti, “Compliance with the: actions ‘Laws ‘of the. ‘United ‘States’. 
SS oreo Ae Studies of State ang ‘Local Adminjstrative, ‘Com- °~ 
hom fs 7  * plidance. and Evaluation Practicgs . for oe 
: (eS Education’ ee a Re 

he “mbes project cis. ne ace it ‘looks. at: 

. ‘dministration, compliange and evaluation at ‘both 

4 ‘a. .- state ahd local levels. Among other things, case 
oS a a studies of. practices in 15 states will. be done as. 
7 -* well as annual reviews and analyses of legis- 
oe fae latively mandated state, and federal reports... eae 


¢ 


) lr es Contractor: | “ABT Awsociabes, tne!” . 
ae a ‘Time Frame: September’ 1978 - August. 1981 


ree Be ‘ana lybis of. Federal ss Framework for Vocational , 2 
oe ae p pene t On a A -_ . 
; —s ' . . - 7 
; is This re will oeua on, ihe. c: suork ween — 
ep oe lf eee Which the: federal government administers voca- - 
os) "0° “tional. education programs by examining not’ ony 
py th the fit: between state and federal legal fram , 

a gs . works but algo the congruence between state ,and | {o 
! " + *° “federal interpretations. of. the laws and regula~ 
: hac} a tions. » The .incentives and sanctions in the * 

a ' federal legislation and.the federal capacity 

a . for implementation of policy (that is, how YS - 
7 oo + various federal agencies relate to state ‘and 
ee a ce ‘Local education agencies and: tq each other). weit 
“2 be examined too. | SO i. 


» 


Os ‘gp . contractor: Request’ for proposal has been issued. 
a = <Time Frame? — March 1979: eenenery: 1981 + +°* . 


“4 » . ‘ . ‘ ' . 
. . . O- ye 


Lobia hited 4 MAP, fil fe e of wwe tg ; 
Ba ee Mate Ags ; 


ea Ce: Ashady of Cates. ang Biforcenes \t ain: Selected, 
oh Rederal, oo : 7 Pa. 


The federa1 experience’ with, coast ade ‘and id enforce- 


as @ . ment over the past. ‘decade ‘in ‘three. other. ‘federal 
 grant-in-aid programs will. be studied’ and: comared:, 
with vocation A education. ee ie ee te a < 
contactor: fay be. done inchouse, by” NIE Ba, 
< Time aol scal Year 1979 ies a rk ee 4 


fe a 


D> ° Vocational Baucation _ ‘cETA coordination a 


- 


Siege BO, og hae This study will” assess ‘the! level and: ‘effects. Of | 
. +4 coordination and\cooperation: .beqween ; vocafional 
a _, education and employment, and training programs . 
out, and identify encouraging ‘and. discouraging factors. 
co . o Information from other on- ab ine studies wily’ 
probably -be used as well interviews and sample ee 
(pes . sae 


: co “Contractor: In the planning process” ag 
my Time Frame: To be determined’ Oe rao 


= - III. Means of Assessing Program Quality and Wetectivenads 
' .A.: State of, the Art en aft Vocational ‘Education 
/ | Evaluation 


“~~ os 
7 = This nendy provides a eavicd of the frees e 
. and a report on the ways states are implementing 
‘the 1976 oo evaluation _ 
; . - . ments.- Problems’ states are encountering in 
a addition to information.on the uses to which - a 
, - ° evaluations are put are also included. ~ 


7 4? Gontesetors CRC Education’ and Human Development, Inc. 
al Time Frame: Complered november mere — 


se OBS Conference: on "Evaluation 7 e 


A conference will be schediied in 1980 ‘to see how 
, - ' states have handled assessments of program quality _ 
~ 4 and effectiveness and to discuss related issues 
a ( and implications for evaluation in the future. 


fo. . F we “Contractoxt to be determined - - - os vo 
Ss ’ - * .Time Fra Fall 5 de - 


i Note: Case studies of evaluation practicés: are being con- 
a oe . @Qucted as part of the ABT projact: Studies of 
Staté and Local Administrative, Compliance and 
‘Evaluation Bractices for Vocational Education 

es te hk OAR ee 
e ; we -.38 a f 


._ =e 


pit gation (C&HE) :Programs:"” -.: ve ae 
oo pla JA Study of the Reaponaivenses, of: tt 
i: Homemaking. ‘Education ; ‘System... coe 


eo ed “This: project whl ‘address several. Sundenantal Lone 
oh eee cerns. of Congress; |: is the intent’ of ‘Congress with: | 
“A “regard to C&HE programs being, carried. out, should : 
' the preserit C&HE ‘system‘be, continued,. and* ‘what, if: 
_negessary, can: be changed to improve responsiveness? 
Data will .be. gathered at the federal, state and 
Po es -. > local levels.. Ten’ states will be involved, five. ‘i . 
v8 es ef which will be in common. with the: other. voca 
| a : tional education peudiee pee. oo. sites ew = 
oe - also be. visited... = . 
a | Contractor: CRC: ‘Education and Human , paveicoeie: Ines x. 
a 7 a Time ‘Frame: September 1978: “ August 1981 ; 


B. Effectiveness of the Consumer and Homemaking Bdu- 
cation | ‘Program | _. 4 : 
4 :@.. 
The» purpose of this seam: is to provide some under~. ° 
standing of the effects of C&HE courses upon séledc-.... 
ted groups of learners paecicestsng in. particukar wg tras 
types of programs. _ 


: Contractor: This study. is still in the eiant Lng as 
. ‘proces’ and will probably: be cond cted 
o~ _in-houge bo 


~ 


: _ | ~ Time Frame: To = determined 


x 


C. ‘Ero jacting What. People will Need to “Know “in 1982 
*, and Beyond: $o be Intelligent Consumers and BE fective 
5 : Homemakers . _e 
A " . : : : e- PS , oe a x 
- A series of panere will be commissioned,- eeaigetad 
_the issues and estimating the kirds,of skills,  - 
_ knowledge’ and abilities different groups of people: 
ey a are likely to need by 1982 and subsequently. a 


i ee - Cantractor: Papers will be commissioned’ by NIE 
- Time Frame: 19,79 


po acest | © ee eg 


i. Dee ‘pies the. ‘gonttactors to che ‘Nation’: Thétitute: Pre 
ot may: ene up aeing more’ original :data: edllection: a first. esoscleg 
', because of. the. limitations and‘délays of other ac ivities: guch. 
cle ag the: Vocational. Educatign Data ‘System: (VEDS)* ae (see. pade 
36), which could have: supplied useful information. : “Other. prob~.:.. "| 
| lems which the study faces’ include. ‘differences ‘in definitions AP ot ae 
across states, the ability: to” aggregate data at the national | a 
level across different data bases, time. frame required, the. lack..: 
. readsty, available data, and coordination of data collection. 


According to.Dr. Loid-ellin Datta the~ intetat sea hiuse aA 
rae voeattonal education field Was, negattve. toward. having. a’ atudy 
wh all, toward NIE ae ‘the. responsible agency and. towand. “tta . 
“vaetanment to the Ed&eation and Work Groupa.22: While vocational 
educators may have opppged the NIE Study originally,.. today they. 
' generally seem much mgre receptive although: some will naturally 
withhold endorsement™ until the findings are ‘released: and. ‘preven. 
to be a ~disposed. a oe i 


: ‘commentac This: sis ination. is pela: directed by © ry David. 
He has~been instrumental in gaining more acceptance ahd greater _ 
trust of “cational educators. He is not a vocational educator ~ 
and is perceived as a neutral but objective investigator. Conse- 
.. quently,‘ Congressional and federal level people have high hopes 
for the NIE Study and-state people are generally less concerned! 
with having to refute the geeaeh oe the study and are more “, 


ae 


inclined to. cooperate with. 4nd support the NIE efforts. The bulk. 

‘of the work for this study is to be done within the next three 

years. The scop gd work as specified by Congress is ambitious 

given the level resources. é mi | ' 
’ . . 4 
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: : a 
‘Lois~ellin Datta, “the New Sonu eeeioaniiy re eee Studies: 
Notes Toward Species Identification," (Paper prepared for 
the American Education Research Association annual conference, 
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ional Senter for’ Educa 


Legislative Basis. 2 | | ee 
“it  ."gea. L6L. (ah (1) The Com tesioner tof: Edugatton) and the ois, 
--Adminietraton of the.National Center for Education. Stattatios 0000" 
. shall, by September 30, 1977,.jotntly develop:information®. «.: Feats 
' elements and uniform -definitiona for a, nattonal vocational»... 
education data reporting and aacounting.eystem. Thies ayatem .. 
 ghall inelude. information reaulting from the evqluativne rem * 
-~qutred to be conducted by section 128 (b) (as sifeh seation will’ -. vm, 
be.in effeet on October 1, 1977) and other information on =) 7 a es 
vdcattional ah an: 8 au vee e, ~ aa ; : le eg te oak - 
(A) students (inoluding information on .thatp race and sexd, so: 
‘(B) programs, ~ — | . oO ee 21am er 
-(€) program completers and leavers, . > 
(D) staff, 
(BE) factlities, and « 
(F} expenditures. 


_ Assumptions/expectations. At. the time of the. hearings held - 

prior to rhe enactment of the 1976’ Education Amendments it became 
-clear that despite the fact: that Congress had mandated evalua | 

. tion and planning and that states had been required to submit - 
considerable amounts of data to the Office of Education, the _ 
information needed for‘ national planning and- policy development 

was still inadequate. Congress wanted and continues to want 
answers td these questigns: . Who is being served in. vocational | 
education programs?“ ttdt-are they: heing'served? What is being — 
accomplished? And what is the cost? Congress was presented: | 


~ 


ue 
e 


tion and justified expenditures and programs.. Many tates had 
individually ,¢ollected more information on vocational education | 
than was required, but hecause of nonstandard definitions and 
terminology, data that were useful at the state and local. levels- 
could not be aggregated at the national level.. Consequently, 
Congress could not be told specifically which vocational. educa~ 
tion programs made a diftershoe andiwhien were costreffective 
across the nation. . ¢ mt eS ‘ 


Throughout the hearings, concern was expressed about a 
need for (1) better accountability of. state expenditures of 
federal funds, and (2) increased state planning ‘that would re- 
sult in program improvement. It was generally felt that’ im- — 
proved data would help both areas. Many witnesses urged the 
creation of a comprehensive and accurate, reporting system, Thug, 


. 
. 


‘ 
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s ot Gangrene: ead ie oe ee ae a \ nigbonaL' at Lea, - 
"WED data ary and ee systenty. Commonly called” ‘ 


eo ease ‘that the. Oyo ten: -be comet nie. with: ‘the’ eee ; 


“vane and State ‘Ocoupational. Information Systems. (see. “page 52. ji 


| "and with the Comprehensive Employment’ and Training Act data sys- 


tem, Congress went on to require that any state. ‘receiving ‘nioney | 


under the Vocational Education ‘Act must ‘supply..information re- "| 


quired by VEDS and must ‘comply. in its Peeore. with" he (axa 


information \ elements and definitions, 


The Amendinents specified that ‘the National Center for’ Rdu-'. 


cation ‘Statistics be given the responsibility. for | operating and” 


“updating the system since. Congress was 80. ‘disappointed with the, hee 
adminieteation of data collection PY the Office ‘of Education, 00. 


It was Congress' intention that "VEDS should be. a ‘comprehen- 


+. give system. Requiring that all local and state agencies, in . 


the same manner, begin counting students, programs. and staff 


using standardized terms was an effort to ensure the beginning ; 


of better planning. In addition, data could’ then be aggregated . 
to the national level. Such informatior would be considerably 
more useful to Congress than any data. previously available’. - 


Members of Congress have long been donessned with follow- 
up--both in terms of seeing to it that state and local educa- 
tion agencies gathered such information as an indication of pro- 


- gram effectiveness and that they then used\| that infgrmation to 


plan for.and improve programs. Requiring|\that follow-up data on “ 
completers and leavers of vocational education programs’ be in- 


cluded in VEDS was’ as much a cémpliance enifotcing. provision as 


a straight data. gathering one. In addition, Congress would hgwve 


outcome data available for use in national planning. Some have 
also viewed VEDS as being beneficial even if it only~serves as 


a onetime measure desining the universe of vocational education. 


Eventually, VEDS data could be used at federal, state, and 


‘local levels in conjunctjon with occupational, information and 
-possiibly result in improved program planning ee sounder policiés. 


-VEDS could also serve as the primary source o 


for Education Statistics. 
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data for other 
groups once the states had Beborcee it to the National ae 
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ae. i, S. Congress ,.House Report of the Committee on Education 


& Labor together with Additional views, The Vocational 

Education and Nationak Institute of Education Amendments 

of 1976, 94th Congress, 2d Session, , 1976, -H. Rept. No. 
94-1085: to accompany H.R. 12835, p. ‘40. 
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sack: avttiv tee! “ithe: ‘VEDS: state’) has’ “worked ‘to’ vac “ommto=: 
8 of the various concerned: groups © ‘and’ still..meet:.;. 
2 tthe: mandated: "requirements. during the ‘initial: planning* and. de’. slop) 
_ ment period. Regional workshops: have ‘been conducted: with : at; te 
' gtaff to coordinate data. gathering: ‘activities for ‘school. res dep 
1979-1980. The’ initial strvey must: include: duch. data as- she a 
kicated counts of the “total cumilative enrohlment in: each: BOR oad 
gram.during: the report: ng year,” and counts by’ ‘sex and-racial/— 
_ (eet designation for. h program, “Other: information: such - ages 
unduplicated counts.of handicapped student enrollment by handi=.". |. 
| capping condition, and limited English-speaking ability and 2.8 ..00°2: 
disadvantaged students by economic and academic alassification 
. will. be Pras requirements in Suture years, - . 2 o . 


the first round of data: are 0 be wathered: ae the -ag79e | 
~ $980 school year. Follow-up data are to be collected in the’ sri, rate ee 
Spring, of 1980 and reported by November 1980. oo Pay se : 


Problems. VEDS has had many eecbiens from.the begdacing: os 
The enormity of the task has been underestimated by. nearly. _—_ 
everyone. Money has been and will continue to be a major prob- 
lem. A very small staff at NCES has had the ‘responsibility of 
putting VEDS together. There does not yet seem to be money for 
analysis, so it remains to be seen whether ° anything can, be Bone: 
with the data once it is anAseeds 


. The real problem of money, of course,’ is with the actual. 
| implementation by the.states. The data requiremepts for VEDS 
imply both a student~based and an. automated accounting system. « . 
While such a (system may be valuable to.states in the long run, 
initially it is difficult. to implement in small schools or rural 
areas. Even states with sophisticated management information 
systems may have to make. costly changes because of. the require- - 
_ment.to use standardized definitions which may be different from 
those used preViously. And since there are limited funds avail= 
‘able for VEDS seep eee ret program meney might have to be 
used. . a ~ 
e 
, , The biggest problems F r tie VEDS staff have ach with 
logistics and clearance. Serving as the primary :data gatherer 
for vocational education ‘has\ meant trying to make accommodations 
for the data needs of a Pacey as of groups such as the Office of. 
Civil Rights, the Bureau of Education of the Handicapped, the ° 
' Urban League and others. ‘ Since ‘data, must be collected and. gub- 
aan by the states, clearance’ through groups such as the 
nt races Department of Health, Education and Welfare (HEW) ,: 
ice of Management and Budget (OMB), and the Chief State 
school Offiter's Committee for Fducation Information Systems 
ale is a difficult and time Qgnsuming EeeR 
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Sogo othe VEeDS' staf¢ has been working ‘to.train state staff and?) 000. 
"to-request: that state boards submit the necessary data: a, 
"year since. the, National Center for: Education Statistics hap been”. 0: o 
"mandated by. Congress to do so. iThe Office of. Management ree ee ee) 
_). Budget has given a-one year approval: to the VEDS forma. «<0. ; 


of" Comments. “The Vocational Education Data: System has re : 
i. ‘geveral Issues: surrounding federal and ‘state/ relations in’ aduca-~... 
oo don. Philosophical questions. and governance are.two such’ \. 9 
ae major issues. “The requirements’to begin counting people and. =” 
-.; . things at the local level seem quite logical and innocuous. \ a ee 
cro...» Suxely local “administrators need such basi ‘information if they 9.6. 200. 
«+ .tare €o seriously undertake. program: Tanning and inprovenent. ) re ee 
“|” However; such a standardized reporting system may Anetta = 
-'needs of Jocal and-state agencies. What ‘ifs: at issue ‘is the \— - 
“ yight of the federal government to require such a system that | 
has implications for state and local poli¢ies,, practices and . A 
resource aldocations. | a Tos (er ae a 


.. ‘The information derived through VED$ could prove to be. . a 
"|. very useful at all levels. It could, for example, serve as a. . 
| formative evaluation tool to determine oe programs. ° 
Some state people have become more supportive of VEDS as they — Ae 2 
| have learned ‘hew they could use it in statewide planning. - 


aeetvemenin 


“The question arises about the necessity to désign the system. 
te count every single vocational education student in this — 
country: as opposed to sampling for the/desired information. 
Such a nationwide follow-up system is unprecedented. 


o- 


National data quality will be limited by the capacity of | 
“states to gather and report 'VEDS information: Some states still 
hand tally numbers from mimeographed sheets.» Qne state estimates 
_ that it would cost three times its total federal grant to - 
implement VEDS. Another figures its data Gathering burden 
will increase more than 400 percerit to comply with VEDS. Some 
states are committed to supportdng VEDS and benefiting from its 
»information base and‘reports. # - we 


' . va . 

 'QOne final concern expressed by some sources is that VBDS 

4“... ° may only doc 't vocational education, not. improve it, at 
least in the short run. ~ a a : 


( 


eg 


Oo") "UBOR = Office. of Evaluation ‘arid: Dissemitiation 


“1. Voeational ducation Equity Study: 6 


..- has been made to reduce.or el{minate such dtearimination and : 
| stereotyping in gueh pragrams\and in the .ccoupattons: for whton 0 
-guoh programa prepare atudenté,! The Commteetoner shall report 3. 7 ots 
date of the enaatment of this Act. 


stereotyping in vocational education prior to enacting the 1976. 


_ standing job of presenting, data to Cangress because the act 


‘preparation for employment. Witnesses in the hearings. conéurred 


- expenditures of funds for the ddvelopment: of non-stereotyped  . }: os 
‘results within two years. The study is ag much a reinforcement 


' yiding information to Congress, reports and handbooks with =. - 


24 


“hegislation Basis. ay a we 
“" "| Seo. 523 (a). The Commiastoner. of Education shall oarry oyt ©. w 
a atudy. of..the extent to which sex discrimination ‘and. sex ateeo- os: 


‘typing eateat tn afl vocational ‘education programa aeatated under” 3%, 


the Vocational Education Act "of 1963, and -of the progress that. 00 


the. results of such study, together with any recommendations ee 
with respect thereto, to the Congrese within two years.after tthe * |. 


ry , 
a. - 


- Assumptions/Expectatioris. The Committe “on Education ‘and ||: o ce) 
Labor held separate hearings on sex discrimination and sex... 


Education Amendments. Evidently the women's groups did an out-. 


includes a whole series of amendments seeking to move state and 
local education agencies towards greater equity. Sdnce sex. . 
stereotyping isa pervasive problem throughout society, Congress. 
was not saying that vocational education was at fault more than = ' 
anybody else. But it was stating that vocational education does. 


have a responsibility in solving societal problems related to. 


that the initiative for overcomgng the widespread discrimination. - 
and stereotyping ,in vocational education would have tq come y 
through new federal legislation.24 Consequently, in addition oa 
to requiring each state board to designate an employee. to coordi- 
nate the state's efforts to eliminate sex bias, authorizing the — 


curriculum materials, and making numerous other prescriptive  —~ 
provisions, Congress charged the Commissioner of Education with 
conducting a study to determine how efforts to eliminate sex .. 
bias and stereotyping have succeeded and to report back on the 


to gee that somethiffg gets done as ide pte research effort to ‘ 
gather information for further policy decisions. .— .° on 3 
- The design of the study is such that, in addition to pro-. 


! 


U.S. Cong-ens, House Committee Report 6n Education and Labor, 
p. , 2 a - 4 Ss he . 


4 . | 1, &* i‘ 2 ie 


, o® 


~ inforbation | to > help reach ‘the ‘goal of sex. a wilt be te 
~ MeLopeS : ace Boia state: and: Loeauses: oe matte 


Vi» The. ‘Equity Study’: is ae ‘by: the.” 
f ion 8: “Off tic ‘of Evaluation: and’ Dissemina- ee 
tion in eee ae a with the’ Bureau of. emtet and Adult: ae soa 


' yEducation. The American Ingtitute for Research (AIR) is the ee 
‘prime contractor and the: firms,.of ‘Stei and Kosecoff, «° 
. ,Ine. and L. Miganda: and Associates, ‘Ines: oueipccetiate del to. 
=” BIR ~ : 


we Oe oes. 4 ee 
ead a The: ‘aboay is. ce to eeviae answers. ue the following. | fee ae 
a -questions: What isthe. ng, of - existing inequities?» To” eae 8 
. what extent do inequitjes exist as a result of policies. ca 
‘practices at. the federAl, state, local -and school. levéls?. a Wt”, da ae 

a’ result of factors in society? How.‘can inequities. be: me a ee 
h. ‘. “nated by efforts df the vocational: community? How much progress. ” 
_ ..has been made in: eliminating inequities? How can. progress. be. . ee 
monitored? oe I - _ eae ee 


; Ot ieaeWeice tent etseion ‘aad extant: data ‘sources were. a oe 

* . analyzed and synthesized to provide baseline.data. State and co 
local surveys. were conducted in addition to hundreds of inter-/ © | | 
views. Thirteen case. studies pfleffective programs are being ae ye a) 

.-  @eveloped: that describe. useful procedures so..that the same’ ee ee 

oe practices can be used in other settings. In addition, instru-  ! 

ments and procedures have been developed and will be included 

in a RepLiyation Handbook that can be used in'assessing sex 

inequity and measuring progress in overcoming ° it. This study 

is nearly onprere: and reports | should bq out in March. 1979. 


Problems. . -According to the project. monitor, eningé ant: ie 4 Ss 
well once they got out in the field -- people were very : 

‘, Cooperative. Their higgest problems were with forms clearance = 

% the process seems to take longer than’ people anticipate. 7 


Conpent. This kind of study may prove to: be useful to, ee 
‘people at federal, state and local levels in that it seeks to. = °°. 
both document the present status/condition of sex equity in — 
‘vocational education and to identify and feed back inform@pion — 
on those things that are working well. However, beoause the 
study is dealing with values rather than simply programs, 
,logistics and other external factors, méasurable results in . 
\program improvement may take a longer period of time before be- 
coming evident aie people would eas ala uve 
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aes Longitudinal, study. ‘on Bild: gual Vocational: Training” 
ot aoe Legislative Basia. oy nae J ees . 
nex ‘18a fa)” The Conmteeioner and the: sdodecany of ‘abew 
together shall ~- 


(1):" develop and- dizecninate’ eccurate information. on ‘the. ee | 
atatue wf bilingual vocational training tn all. parte: of. the Be a 


ae 


United Statea; © . = uae el 
(2) evaluate the ‘impact of: auch bilingual’ soeatienal: 7 15 
* bnatning on the shorta es of well-trained peradnnel, ee 

o unempdyment> or underemployment ‘persone with Limited 


|, English-speaking ability, and theability of guch persone ae 
acquire suffictent job’ skills: and English “language ekille.. ie 
_~ > eontrtbute fully to the economy of the United Statee;.and 
(3): neport their findings annually to: the Preatdent’ and the 
Congress. ; ; oo. .e oa 


(b) The Gonmbactone. shall soniye: with the Seoretary of “ 
Labor wtth respect to the administratton of thia part. ae o 
. * °. Regulations and guidelines promulgated by the Commissioner + 

_. tp earry out this part shall be constetent with those pro~ 
{ * mulgated by the Secretary of Labor pursuant to section 301 
(b) of the Comprehensive Employment and Training Act of: 1973 
and shall be approved by the Secretary a ae Lt " 
tésuance. | 


-Assumptions/Ex sctations..° The. first étcest Waeiglacionee 

designate persons. o mited English-speaking. ability as a. 
specific target population for training was.the Comprehensive: 
Employment and Training Act of 1973 (CETA}.. The_Congress felt. 
that mamy of:.the thousands of unemployed and underemployed. - 
limited-Enghish speakers could also benefit from vocational 
training. Thus, Part J -.Bilingual Vocational Training was 

- added to the Vocational Education Act by the Education Amend- 
ments of 1974. The Education Amendments of 1976 retained bi- - 

_ lingual vocational’ training as a separate categorical program - 
since it had barely begun to be implementé@d and required states. 
to allocate’ 20 percent of vocational education funds to tha, dis- 
advantaged and persons of limited English-speaking ability. 


- The section above: that specifies an annual: report | onthe. 

- status of. bilingual vacational training and an evaluation of the -. 

". impact of such training was included as much. to-insure. that: the 
Commissioner of Education. would address his. attention to the 
‘programs as to actually provide: information to Congress... Clearly - 
it was a priority -item and Congress did not want sl +gnored Or. 
lost ‘in the myriad of other programs. 


v 


|. @f training they. could benefit from -mo 


_' -a@regs'can be measured over time, as. well as providing. an aa 


2 ee evaluation. . ae pe & aes = 


viet —s 
eas a ee ‘ he ve 3 ey = ere 
one or shee , “ 


ability -+ how many. are unemployed or\underemployed, what, kinds. 


“as a result of: training. ‘Longitudinal ‘sumveys, were ‘set: ‘up for 
“both CETA’ participants and vocational, education trainees. “These: 
evaluations’ should provide a ‘base of information from which. ‘pro-" 


of outcome variables.’ The vocational education study’ ‘should 
provide data to,Congress that will be useful in’ making» policy, 
. decisions. - \ Program. improvement at the local’ level may come: 
nePeue simply because the’ study iad being done. ' ee 

| * Approach/activitdes. The U; S. Office: of ‘Baucation is. 
-gponsoring t Longitudinal Study of “Bilingual. Vocational - 
Training and has contracted with RETeehae es Inc. ta do. this 


mY ” 


. A number of projects have Beau set. up since 1974 that: pro- Pane 
vide vocational’ training through the medium of both English. and: | 
‘non-English languages ;. Enylish ‘Language instruction is ‘provided — 
_ for those who need it. “The: longitudinal study is examining the. ° 
--affeets of this: ‘bilingual vocational training on the work . 
experience of trainees, The effects will be méasured ‘primarily. 
in-terms of changes in employment: status and earnings by looking . 
at Pre- and post-training labor. force’ a aa employment 
and mages over time. - _— | neu! 
A probability sample of gepkou meaty l, 000 limited. 
Jish~speaking. trainees yas selected from bilingual vocational . 
eening classes idéntift ‘thréagh an inventory conducted during 
ae fall of 1977. .Trainee& were randomly sampled from each of | 
the 136 training classes in the sample universe and dee located 


in fourteen states. Interviews were conducted with trainees, 
: instructors arid program directors, Basic descriptive data on 


trainees Paes lasses and instructors are also bein 
sai tabulated this year. | | gq gathered 


the first round of interviews has been completed and 
oo interviews wil be conducted in spring 1879. 
Lok roblems. The major ars this study enceuntered was 


“. finding vocational programs that’ were truly pat inguat as defined 
“in pele Uo ~482. ; 


Comments. As stated cesvioddiy, this study is as — a. 
compliance | enforcing and monitoring device as it is. an improve- 


ment-oriented evaluation. ‘Like many etudice that focus on hard. ote 
“- “ hs \ : t H 
& . | \ oe 
a ‘ 
ry , ‘ 
, ap 4 
: \ ‘ Ory 


“Little is known’ abot: : people ees ie Be 


t; what: happens. to them | . : F ans 


y * 


\ 
hy, 
NS, 


re ouegome data’ ar measured ‘in .the ! : nm ae 
Toil. “1 BQt be well: received ay some in the field: unless: some attempt... 
ve. *.  dsvmade.to”add ‘some contextual ‘data. Some hope that ‘this study. 
"will be continued with future fgllow-ups so:‘that it could become”: ” 
0. truly. longitudinal ‘and provide’ data on long-term effects.) 9), 
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3, "Models for state Use in Eva}uation. Vocational-Education’. 


“ments of 1976, it was. undestaken to provide the framework of an: 


/\ evaluation system which states. could wae. to comply with\the 


page. 15). 


4 Adult 


* . 7 


\ evaluation: requirements specified for them in that Act (see |” 


et in conjunction with the Bureau of Occupational’ and: ~ | 


velopment, Ing, to develop the models. = : 
aa Within two.years, three models -~ process, impact and out-_ 
come =~ were\to be developed and field, tested. “Then a State _ 
- Evaluation Guide. describing the models and how to.‘use them was. 
to be pyt tagether for the uge of. interested ‘states. a er ee 


~Thia project was cancelled in October 1978 when it became .- 
apparent that it.was impossible to complete the work with the 
time and funds available. It is not yet ‘known whether any kind 
of a report will be issued with the. information thus far 
.or whether anyone @lse'will be funded to také 


r an where this. 
project left off. lee o , a oe 


a db ae 
ty 


“Labor market,” final reports: may. °: 


a: cast . 
of Le mt 


While this project was notj set up under the-Education Amend-’” 


= sagite Office of Evaluation and Dissemination sponsored this >.” 
_ proj _— 


ae 


ducation and contragted with CRC Education and Human pe- 


gathered 


/ interest to.the rere peer y: because what ‘they. ion fay” hae see 
~an- impact on present and” futurg evaluations, Numerous telephone ag 

' - Calls, were again made to get..a status: report’ on current activities...— 

The particular groups’ are’ included: to provide a. view. of the. ange, 

aad Bobry ities eo a compe chensive). views” 7 
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National Center _ eer’ Research ain Vocational § Education 


The National Center’ ‘for Research ‘in Vacational Education’. was a 
. also. created’ under the Education Amendments of. 1976.° ‘Aihong. numer= =. °° 
ous.other activities the Center is to work with states, Locak = 9 8 
educationa& agencies and other public agencies an devekoping PAS Te ie 
methods. of evaluating programs, including the folLow-up studies” a 
_ of program completers and Leavers required by. section 112, s0 that. 2°. 
“ these agencies can. offer job training programs which Ane mone . a re 
closely rekated to the types of jobs available An win communt- 9 et, 
-,, thes, negdions and states (Sec. 171 (a) (2) (£)).{ The. Ohio State _ re 
“University received a Gontrect to SPEEane: the Center for fives 0 0 0 Mis 
years, through | isk ; o. Me a . 


“Through: all of its services, “ineluding dsesieseiy ‘related 
to evaluation, the National Center is intended by Congress to 
contribute to the improvement: of vocational education ‘and make 
it a'ttore relevant and flexible system of education. ~The Center. 
is also to serve as a focal point for assisting-state and: local . 

- administrato?s and. educators in iii ae the eaPEreise needed 
we conduct evaluations. ¢ ee, . a 

During their first year, the Evaluation Division of thes _ *S 
National Center focused on providing four states with technical _. 
-evaldation assistance -related to- follow-up studies of completers 
and leavers and special needs subpopulations. Case study reports 
and handbooks for practitioners are being prepaxed based on the 
technical: assistance experience. Another project includes pro~ — eee 
viding assistance to the developers of evanuaeroy reports: ko in= | 
crease ithe quality of» Enpee reports. . 7 

e ’ 

‘The “National Center. contracted for a number of papers. as part 
of a state-of-the-art monograph series. The following are direct~ 
Ly. related to evaluation and should be available in early 1979: 

’ Needs Assessment in Vocational Education; Longitudinal, Methods 
for Evaluating Vocational Education; Use. of Evaluative Data by: 
. Vocational Educators; Impact Evereepnen Data Sources for Voca- 
aot sonet Education Evaluation. 
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«ol the bther projects at, the National..center that will have.an . 00. 
. |, ‘impact onthe pole nar 9 of vocational education are Examining . : 
a Vakational Education Qdtcemes and their Cornretates and se appa . 
Outcome Measures in Voaationad Education (funded. under a evious .-- 
ny “contract with NIE). = | a oe > a ye 
* “During 1979, the National Center igs proposing to continue pro- 
viding technical assist nce to the four states previously worked  —- 
4, ‘.. with, concentrating on /the evaluation of programs for special 
‘| _ needs subpopulations. /They are also proposing to-develop -speci-~ . a 
‘fications for a longifudinal ‘study of former vocational education =: 
students, to datermi e the correlates of placement rates in voca~’ ,. 
Pine re tional education) to continue work on dutcome studies and process « 
variables, and to develop more evaluation handbooks. Proposed 
ae, -handbooks include a guide“for evaluation ‘by. state advisory coun- — 
a cils, and examination of the case study approach for evaluating. 
: vocational education programs, and’ some strategies for measuring 
pS performance skills’. _ us Aoi 
: et While the National Center, is not one of the many agencies 
i‘ “ directly involved in ‘evaluating vocational education, the pro- 
; “ cedures it develops and the services it, provides could have a - 
great impact.on*how evaluation is actually done in the fteld. 
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ite ement ing the s¥stem and they are getting upstate occupational 
| uy,  idtormation coopd: Ming committees (SOICC's): to meet the commen 
“speeds fon the plaming for, and the operation of; programs of the 
‘state boaid assisted-under this Act and of the-administering agen- 
eked under the Comprehensive Employment: and TAdining: Aet of 1973 . . 
“ (Sec. 16% (b) (2)). It is. clear that in mandating: these. committees 
and allocating funds for theiy use, Congress was responding to the -° 
. numerous ‘complaiggs that states were not planning vocational edu- 
‘cation programs in the light. of current and projected job démand - 
2 data and other complaints that available demand data were-in-. - . 
_ 4. auequate for planning, particularly at the’ local level. _ 


a? local levels. The states dip reac aadl hiaaetipen anne carn ‘a 


_ | NoI¢c and the SOICC's have. the potential for providing a... 
. basis for improved program ‘planning as Sell as bringing. together = 
_/ ° Department of Health, Education,’ and Welfare and Department of © 
Ly, Labor people to agree on coordinated data. ‘Since NOICC and the 
“ .. §OIcc!s are not: primary data gatherers and are ‘seeking to-avoid . 
_ duplication; they are in the position ‘of coordinating and improv~- 
aS eee dialogue and coopefation between the various,agencies inw » 
“ee olved in the delivery of education and training Sefvices. - The’ °- ’ 
plan is té feed occupational supply data into a model from a 
_ variety .of sources including the Vocational Education Pata System 
__ (VEDS) and the. Comprehensive Employment and Training Act (CETA) 
programs... Eventually, the system is envisioned to include data - 
‘an occupdétional demand and supply based on uniform definitions, 
_standardized estimating pracedures and standardized occupational 
classifications. Fo the first time, it will be possible to plan 
training’ programs frém the state level based on a fairly acguvate ~ 
picture of the demand, for an occupation and a more complete view.,~ 
-of the supply within a state. “tn other words, planners would be” - | 
able to look at, the total number of people being trained as nurses ; 
aides within the state rather than just compare that with projected 
demand figures befére deciding whether to add, subtrac , or ‘thain- 
tain nurses aide programs. How decisions are to be made regarding . 
which groups should offer whigh training programs rethains to be - - 
=) seeni At least they are beginning to talk. a eo a 
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Bee in ‘ene: ‘future ‘the: state foe tee may: Daye: a deoat ns inti uenge 
: eur “tha evaluation of voqational_education.. Data ‘from the BOICC's 
'." gould: be key ‘dn evaluating ‘the planning: process and “in: establish- 


. dng | rogram: goals.’ As more: emphasis is Placed on outcomes; | the. 
placement / f students’ could be looked ‘at..within: the: context wf. 
“state and’ local labor market conditions. ratfier ‘than as “simply | coud. 


upbers.. Local labor markets are both open and. in a continual © 
JfAfux, so short run evaluations of placemenf will never teil, the 
.~ entire gtory.. Placement figures must also begin to reflect such 3.) 
. things. as whether vocational education graduates displaced. Other (o's: 
‘workers. The issues. surrounding sdcial. good maké, Clear. measures ee 
- of success difficult at. best. However, BOICC’s #hould at least . ee 
begin to provide. substantially better data than have previously oe 
been available which can be' used in the planning: and evaluation = ° . 
of vocational education-at.the state: level. SOICC's. may eventu~ . 
ally help determine which training programs..are continyed:and "70 | 
which are discontinyed, while at the same time getting ‘education ae 
and human resources personnel to. work together. oo 
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2"... “he General ‘Accounting Office (c 
“vocational éducation (see, for example, ‘the reports entitled 

Taadning America's Labor Forces 
ational ‘Education issued 


1974), hag had “a 
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| ctober 1972, and What 44 "the RoLe. 70 
Qual Assidtance for Vocational Education? issued December, . ee, 
great impact’on the way that the evaluation rew ° 3: 
quirements ‘for vocational ‘education now look.” Congress: has gen~ .- oe Lae a yoy 
listened to the recommendations made: by the’GAO; and the oo..." 
Al education “community will be’ very interested‘in any* ||: 
reviews on vocational ‘education ‘the office might undertake. 9 
' AS thig writing, -GAO had’ not yet ‘stated ‘a déecisio 
any additional studies. | They will be gathering data 
is’doing-what in the ongoing evaluations ‘before decid 
ig a need for them to conduct a sélf-initiated study or 
cient information can be caiman tig sources. ' One of |. 
‘their greatest concerns, of. course, -is wi 
dollars are being spent at the project level.’ The National Insti- 
tute of Education study on the distribution of funds (see page 36 ) 
may address this issue enough: to satisfy GAO needs. 
whether or not to conduct a study will probably be made’ in early” 
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sible for the administration of the Vocational Education Act el 

pac. 160'(a)). Many of their activities. affect the ‘evaluations noe. 
one at the state level. They conduct. the Mahagement: Evaluatiqn. 9290: 
Reviews for Compliance and, Quality (see Page 28°) }. provide: Anter= 90 
pretations and policy statements on the federal rules and Yegula-. Lae 
. tions; review, analyze, a ahd. approve five~yéar plans, annual: program © om 
plans. and .annual accouhtability A ape and: ‘provide ‘technical, 
assistance to the states. In addition, they are able to. contract 
for studies that will help them in their administrative Hasire 
For example, in the fall of 1978, :a small study in nine states. 
was-.being conducted to ascertain the cumulative data burden-:on ..=« 
‘the states in complying with _the meee planning process.". 


“The lack of a permanent’ Deputy. Sonlesionas (until November | 
1978) may have limited the Bureau's effectiveness with regard to. 
- evaluation’ since the 1976 Amendments were passed. . States need 
‘and want guidance that is not restrictive and there is general - 
aah ‘that things are beginning to improve. — 
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th Ne Coordinating Conmittes a on Rose ech’ in. Vocational 2 mdugntion © 


“Bee.. 471° (B) (4). of the: ‘Veoationad’ nducation’ Notas. ‘amended: ee 
a a 1976- established this committee within: the ‘Education Division’. ©, 
| jiof the. Department of Health, Education.:and Welfare with oe 

. ship consisting: of. representatives © fromthe National: Institute’ 
...9£ Education, @ffice’ of Education. ang: ‘Pund. for. the’ ‘Improvement ee soup ah eed | 
-...  *0£ Postsecondary: Education. Representatives from the Bureau of 0 Ue 
-:' Occupational ‘and Adult. Edugation: “and. the Offide of Career Educa- ee ee 
"', | tion are also. on the committee. This committee is to.dévelop a. ae, ae 
plan foreach fiscal year; establishing: priorities for the use of | 
_ federal funds for vocatiénal. educatien research, career. education . 
.-research,, education and work research, exemplary programs, and. 
curriculum development. The committee is also to develop an ef~' 
eeeriNe management information system on. een ere i nee 


‘Sos@®ar the Committee has met’ once. or twice ‘a month primarily. 
to share inforf[ation and keep abreast of what the others are. 
doing. - They have contracted with the National Center for Research 
- in Vocatiogial Education’ to prepare and publish an annual report 
* and a: mid-year update that list the federally-funded args in 
| progress related to wecets One? oun cans 
a It is hoped. that this committee wilt help ‘eliminate linnecessary 
dupligation of research: projects, systematically collect informa~" 
tion On research needs and oversee some kind of quality control - 
gn research performance. Cangress outlined some priorities for . -° 
research and apparently would like this committee to see that — 
those are followed and that research efforts become more account- 
able. They may also see to it that research-needs related to: — ce 
evaluation are more systematically addressed’ in the future. — . ro 
4, ‘ ; : . : : 
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‘Miscellaneous ; 
There are numerous other efferts that stould-at least be men~ 
tioned. The National Longitudinal Study, the National Assessment 
of Educational Progress' Career and ‘Qccupational Development ° ‘ 
Survey and the National Center for Educatioh Statistics Class of 
1972 study, are ongoing ‘and elicit national information. The 
National Center for Research in Vocational Education is. respon- ' 
sible for gathering together the infOrmation that Project Baseline 
once assembled. That project i8 called National Vocational Edu- : 
cation Needs. In addition, numerous states' research fpordjinat- , 
ing units (RCU's) conguce or commission studies SEetares to evalua- 
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~ & The reader, by: noiw, ‘may be wondastag: i, lui all “dyed 
evaluations going oh, anyone has time for anything else an ates 
whether. anyone has. an. overall conception of how ajl ‘the evaluas . 
; - tions fit together. “Thegée :are not, all of he agencies. asking. 
r ‘for information. There“are accreditation. revi ot hae vocational 
program approval assessments, and Office of Ciy 1. Rights. purveys” 
. . to mention a few. There is a burgen placed ‘on state and local | oe 
"  #) s education agencies who cooperate in térms of both providing. oo 
“time and resources. ‘These states are hopeful/that vocational ees 
. - education evaluations will also Pore ‘useful. information for” 
PE eS ‘program amprOVenenys . e 


While there is general aeseawan’ that che. ‘dntest of the. 
1976 law is fame it may: have. been a bit unrealistic pr too’ | 
optimistic. "Much of what is going on in: the name of. evaluation. a 
.is done in good faith, but it ib done to comply. That has as oe: 
much to do with the state of the art-of avaluation ‘aijwith the ae 
manner in which things are.evéluated. -Doing a follow-up of .. ea 
stifdents and employers does not necessarily tell the local’ 
@ducation agency which things need to be changed in order to 
,  , . dmprove the program. am. Evaluation, like, testing, can affect 
\ | the priority assigned to tasks. Because much emphasis is placdd. 
‘on follow-up in ‘the legislatidn, it can be hypothesized that 
state and local education agen néies.. may’ choose to concentrate, 
their efforts on building go d fgpiow-up rather than on ‘exploring | 
apecseaG ways to improve progr 
Vocational educators are. genérally ' realists and many. see 
‘the importance of evaluation data they are now being asked to” 
collect in relation tg the "vogat tonal education legislation ~ 
‘ jn 1982. The ‘numerous requirements and lack of funding for. . 
‘evaluation may have the net result of compldte compliance but © 
. insufficient program improvement. The incentives to do more 
«than epmply are not altogether clear. Those states that’ de- 
\ veloped extensive follow-up systems in response to localsneeds - 
: ‘ "may now have te spend: considerable time and money reworking — 
-*. + gygtems to meet the requivements.of the Education Amendments 
‘e°~~”s*=<«é‘ | 1976... Monsy 8 rarely. withdrawn, because of poor evaluations. 
\ '* In fact, groups with poor evaluations may be the ones’ that need 
: | assistance anes, RS 


@.: 


; *% 4 * \ e aby. 
a we tates a j 
F569 WES SBT 
te % bos 


. : ¢ et ee 
3 ‘ 
: . : ve oe 
vos ns . i , ’ ‘ 
rr ie + Cah oa e ra 3 * o. 
F to oat Poti ees! eae tg pe ae Mean 8 i er 
; : f ee eee tin Hae : 
nN we fh we, gs a _ .? 
a a, ey ; . ’ ay . 
ga ag : 4 tf , 2 , 4 ai on 7 
f : oe o 4 a . ¥ , 
: * =m wy 
. e 


ie piimary’ purpose of Pe laec ion! is prepean. Anpeauenene 


at the loca] level.. 


_.it possible that. such ‘evaluations ean serve | 
- or eattier group... at the nene time? # 


x 


me} “Are ye ‘reaching a- point of. diminish 
‘eregard to the amount of evaluatign ! 
. dmprovenent? ‘The cumulative effect. 


den and cost may become prohibitive. . 


Is 
the ners of scaled 


zo “te 


ng. ‘returns with. 
\vis-a-vis" program \ 
in aa of ae 


© Does anyone know ‘what the real dncantives are. for 


compliance? ; 


Oo: * M9 what extent "ab" geaerad evaluation — 


re state and local policies? 


a 


an “groups? 
a F 


o What provision is theré to pull phe 


a 


iy 


6. Is there any nOee for interface sad these various oe 


poeeneialiy con- ga 


ron 


.flicting results of all these evaluations together 
into one coherent picture of vocational education? 


lated into reality? * 


AY 


How well, is what Congecee wanted done being trans 


* 


a : , 
_ At some’ point vogational educators at the local, state 
and federal levels as well. as members of Congress, must define 


' their real information needs in order t 
vocational education. 
addressed’ separately. 
#ions' will be less than optimal and we will 


actually imprpve _ 
Questions of accountability must be 
Until that is done, the ongoing evalua- 


continue to. have 


a situation where too much is still not enough. ‘ 


’ 


: “iAnd' yet, concern is. expressed: ‘throughout a 
’ the legisla ion about - ‘also being ,able to. use evaluation data 4° * 
‘ to aid, decision makers particularly at the federal’ level. 


In, wohiecting on all. of ehies: more: questions | aueeare than _ 
answer but. ce seem to ‘capture some. bine ta themes: . oe 


T. 


ge Lp they 
. a a . 
oo te ee : : ee jek 4 Fi non Bs os 


' The. eduGabion ‘Anendments of “1976. snoiuded” ‘humerous requires’ acy ae 
- wments for the dvaluation. of vocational education. This paper has . 

provided an overview of the aur and the ere charged | re : 
“with evaluation activities < ; a fae Fe a 


“All of*the groups are Seagate ad. to: the aanigeans: some. with’ 
more ‘enthisiasm than others. Few people question the intent, ef. 
‘the evaluations; they sometimes. question how to pecoeed: , 

9 : 
[There are some fandamental problems, however: Zz . ne 
sy ee : . . ae 

o Many etate boards | of: vocational sducatlon feel that they” oo 

are being asked for too-much too soon. They are’ not: al-: 

ways clear about ‘the incentives-to comply. It is not known. “". 
‘ifthe methods they use for aUeTNaeion will ene help | oe ‘ 
_, improve progran NS « . P Ve 


re) State advisor eencetie for vocational education. ‘scmetines 
‘gee their goals as incompatible. Acting as.advocates while: 

oe ‘ evaluating the total. state program ‘of: vocationad | education. 

. % | ds a potentiag conflict situation. | §till, many councils = — 

os ; He that they must have the power to evaluate or asses — 
iff/they are to offer appropriate and-helpful advice. | 

’ lay pdople, se aa aeey technical assistance in orde 
evaluate. 


ce] 


‘The National Ra tase Soin for Vocational spaak blah: hae 
‘the saine conflict as the state councils with regard to ce 
evaluation. A addition, . its mandate to evaluate ‘is not” fr 
, altogether: clear. : - Cds 
a ek 


‘The Management evaluation a ciae and many: | 
conducted by the Bureau of Occupational and Adult Education, . 
does evaluate compliance'on paper, but no one is sure if. 
there is a relationship between such Corps ttiee and ae 
| improvement ;. SO es ‘ 
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f:@ The National tistitute of Sanéation ‘s “ehovougi, sendy and | * 

" valuation of vocational education will contribute to a 
-4  ° broader understanding of vocational : ‘education, ‘but it in: : 
| "ambitious, qr: the level of funding: ; 


’ % 


ae ‘Vac ational ‘ pduenedion: Data: soca ‘bat wet up: py ‘the; ae 
Wational’ Center for. Education Statistics. has. encountered:.. cae 
‘fundamental: problems’ in: ‘the area of state and ‘federal, ten \. 
lat&ionships. In. addition, “the scope of, a nationwide -d Fee 
ayaten in: which everything Rex counted °4 is alfgieult: ta, come on f 
-prehegd ‘4 ' . . 


the studies are So chuctek — ‘the: Office ‘of ‘Byaluatton.” 

“and Dissemination on sex. equity and ‘bilingual. vocational - 

education.are steps “foward: } addxesqing some 2 major: probleris ; 
in the. field. | re 


- 


rare 


The National Center. for Aecearcn’ in vocational: Haucation. 


ig able to provide technical assistance ee nly. Secs iG 
four states with, regard: to evaluation.’ ‘ . eo le be, 


National and: state occupational infofmation goetdiaae gis 
committees that can. function well will take years to de-. . 
velop. In.some- states” there ‘is. little data. presently heing 
gathered that . they. can. coordinate - for. esse -to Ws» 


The General Accounting. Of fice has not provided: Peedbacic: on. 
how well everyone is. proceeding with, meeting ‘the mandates . 


eerste 


Phe Bureau of: ‘Occupational and Adult Education haa. a “Leader-"" 

_ Ship role in evaluation which is complex, - challenging, and 
“not yet fully | peetheee : _ 
The Coordinating Committee on. Reasearch in iédaticnal ‘Edu- 
cation is only beginning to function and its: eueh saa 
is pet yet Known: “4 ep 


Numerous. sehee groups’ are “giwe acuta with evaluation 
and one. can only ‘wonder if figures from one study. are ever. 
compared with those from another. At some point the in- 

formation from all of these: evaluation activities must be 
_ integrated. a | 


Everyone seems to want a respansive vocational education sys~ 
tem that has realistic goals and gets the best results - for the 
- dollars spent. With these kinds of ohjectives, however, goals 
for voGational education must be more clearly stated and appro- . 
priate criteria. for measuring success and failure must be estab~ 
lished.. In addition, duplication must be avoided not only in 
program offerings but also in, evaluation. 


Although a system of checks and palenees May be useful in: 
order to assure compliance, it may not result in program improve~ 
ments that are proportional to the dollars and’ energies’ expended. 

on evaluation. Many people, remain optimistic that the problems | " 
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| ar « vege Cichel: edide’ ion. are: bettag. ‘addr osed ‘through “qvalue ion, oe 
.., but: question when, or at. what point, pit create. MAES, ae 
op ahuees than. they oe aes 
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4G ps it, Ws tout 


AS the different groups mentic od: in. ‘this paper. ‘gain exper", oe 
-dence and hecome more aware of each other's evaluation responsi-” a 
' bilitges, - they may. find ways: to. werk ‘together » to avold duplicas 00... 
_ tion Phile still remaining in ‘ca liance with legislative prom. oe 

visiong. until/if they are. changed. ° ‘There needs also to be a. a tes 
contin dng effort to return -the':-results of evaluations to people act. 
. be the: local level’ of qubatantive. a silage are to. * ate 
be made. ” i - | oar 


' The. federal goverment has acted. ‘in’ a leadership role an me eS ae 
vocational .aducation since the early 1900s. Because.the overall. . 
impact. of federal influence has been positive, ¥ ew. in the voca-. - ae 
~. tional education community have: questioned the"¥ederal preroga~ |. -! 
tive to demand a great deal in the way of compliance, data and =}. 
‘paperwork, despite the fact that over eight dollars of every ten. eee 
dollars spend in vocational education: comes from local and state — 

. sdurces. The majority of vocational educators. feel that the. a 
evaluation: requirements in the Educa@fion Amendments of 1976 have — 
been highly beneficial as outstanding efforts have been’ made ( j pees 
acrogs the country. to. comply” ‘with ch requirements. A oe 
It is clear that Congress mandated strong évaluation Ye~ ; 
-. guirements “becaus¢ of its real c@ncern with vocational education. ' 

- Vocational education is indeed beginning to change and <improve,.. 
but it is difficult to know if the cause is the mandates or a 
generally changing consciousness. State and local education 
-agencies need to know soon, however, that they‘ are doing the — oe 
right things and are on the right track if.they are to show the 4 _ 
aad of improvement that Congress is seeking’ by 1982. 


Few states have formulated. comprehensive iooaticasl eauee: { a 
tioh policies. As those policies are developed, in the dight of § ---~ 
incoming evaluation information, it will then-be possible to con sys 
“duct ‘further evaluations in accordance with them. State poli- / 
. ¢les could ‘increasingly become a corsideration.in the develop- / 
athe of federal policies. and legislation as that takes place. 


. ‘As state and ,national. vocational etauatien ‘policies are. 
. developed and refined, it will become increasingly possible for. 
the vocational education. community to formulate the questions 

that could and. should be eee and then answered naeee évalu~ 
ation efforts. ; . 


The Raueatdod ‘Comins waite’ ‘of ‘thie states ‘ence, “an cooperation - 
with. the National ‘center for Research’.in -Vocatiqnal Education : 
_ prepared a ‘paper ‘entitled: "Evaluation. of ‘Vocational Education: | 
Roles, Responsibilities: and Responses of State and Federal ° 
Agencies." The. paper, which was developed between June 1978. and. 
- January 1979,'was intended ‘to provide | an: haceua overview of - 
“current evaluation activities. : ms 


: The National Center hosted. a o saikavenaa: on’ ‘the evaiuation’ ae a. 

ae - of vocatjonal education of practioner in rvocational edu¢a- — 

*  ¢dan on anuary 4:and'5,° 1979.. A draft of. the | ECS paper. was. 
used ag.A basis.of discussion and the: group was asked to sonidos 
and propose ‘some initial concerns: and/or tecommendations relating © 
to evaluation. This is a synopsis of-two full days‘of. discuasion -. 
by fotx executive directors o ate advisory ° ‘councils for voca- .—. 
tional education,. three’ state eosational: administrators, four, ‘state « 
evaluation coordinators, one National. Advisory Council for’ Voga-~ fa 

. tional Education member, one state occupational. information cdor- ~ 
dinating committee executive director, two ECS: staff members and , 
two National Center staff et A meS of phe cetee ee is n= a 
Cluded in ‘the Appendix. 


Fr 


‘t 


Participants were asked to share their concerns about the. 
-evalugtion of vocational. education and the group then. moved tov 
, formulating recommendations. The intention was to go beyond the a 
. ustial rec ndations for increased funding for evaluation eff rtey 
o There was ttuch discussion and debate and gitle divergence of | 

; opinion--but consensus was reached on at: least two items. 


oe , din evaluation. While the tion reqyirgments art ee ce 
\ . . fied in the Education Amendments of -1976 do. present es pale 
ee Sg . lems, there have been significant benefit oe 
a | , improving, and stimulating vocational eficathen. Federal '- ar 
a pressure has helped establish an abpro riate jority, on iu c. 
aa mnpeugneut the vocational education syst : ee 


© The group  exprasaed: its endorgen ement: of - 


on eyaiieticn to, receive and ecoradnate data from all on= 
going evaluations. should be established or designated. — A; 


~ 


; spdepee | ‘having - a. eae positing te A about the fe eral’ 7 
..! evaluation. requirements and. ‘thé Q idireétians ‘for: avaluation-. 

a vocatiorial education, individuals Within the group ‘also: had a... 

, Rumber -of @oncerns ‘about ae This section: deh race those! 

ecg La - Cana: . oo oy fe 


- purpose 6f evalua ion. There. was sandanehtay- concern, about >. = 
cne-Feat an ipl: purposes of evaluation in the federal ‘voca-" | 
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